Training Vision-Language Transformers from Captions
Authors: Liangke Gui
Yingshan Chang
Qiuyuan Huang
Subhojit Som
Alexander G Hauptmann
Jianfeng Gao
Yonatan Bisk
Venue: Accepted by TMLR ()

Abstract: Vision-Language Transformers can be learned without low-level human labels (e.g. class labels, bounding boxes, etc). Existing work, whether explicitly utilizing bounding boxes (Chen et al., 2020b; Tan & Bansal, 2019; Lu et al., 2019) or patches (Kim et al., 2021), assumes that the visual backbone must first be trained on ImageNet (Russakovsky et al., 2015) class prediction before being integrated into a multimodal linguistic pipeline. We show that this is not necessary and introduce a new model Vision-Language from Captions (VLC) built on top of Masked Auto-Encoders (He et al., 2022) that does not require this supervision. We seek to provide general advice on multimodal pretraining by examining the roles of (a) unimodal initialization, (b) unimodal architectural components and (c) data annotation in the pretraining corpus. Our extensive and carefully controlled studies suggest that none of the above factors is absolutely important in achieving versatile vision-language representations. We conclude our analysis with suggestions on the choices of initialization, architectural components, and annotation formats targeting a better balance between data efficiency and representation quality.

Published in Transactions on Machine Learning Research (09/2023)
Training Vision-Languaal Transformers from Captions
Liangke Gui
Yingshan Chang"
Qiuyuan Huang²
Subhojit Somè
Alex Hauptmann'
Jianfeng Gao?
Yonatan Bisk'
'Carnegie Mellon University
Microsoft Research
gtthub.com/uuikk/IIc
Reviewed on OpenReview: http:://pppareeroe...ett/rrrrrrp
Abstract
Vision-Language Transformers can be learned without low-level human labels (e.g. class labels, bounding boxes, etc). Existing work, whether explicitly utilizing bounding boxes (Chen et al., 2020b; Tan & Bansal, 2019; Lu et al., 2019) or patches (Kim et al., 2021), assumes that the visual backbone must first be trained on ImageNet (Russakovsky et al., 2015) class prediction before being integrated into a multimodal linguistic pipeline. We show that this not necessary and introduce new model Vision-Languagy from Captions (VLC) built on top of Masked supervision. We soek to provide general advice multimodal pretraining by examining the roles of (a) unimodal initialization, (b) unimodal architectural components and (c) data annotation in the pretraining corpus. Our extensive and carefully controlled studies suggest that none of the above factors is absolutely important in achieving versatile vision-language representations. We conclude our analysis with suggestions on the choices of initialization, architectural components, and annotation formats largeting a better balance between data. efficiency and representation quality.
(He et al., 2022) that does not require this
A pitcher at a baseball game who has just thrown the ball.
COCO annotations
1 Introduction
Should vision guide language understanding does language structure visual represcntations? Vision- language (VL) transformers have put language first. Most popular vision-language transformers (Chen et al., 2020b; Tan & Bansal, 2019; Li et al., 2019; Lu et al., 2019) only integrate vision from se- lected bounding boxes extracted by pretrained Tm- ageNet (Russakovsky et al., 2015) classifiers. In this paradigm, the bag of visual lokens are embedded into an existing linguistic space (i.e., the lexical em- beddings of BERT (Devlin et al., 2019)).
The intrutccioon of ViT (Dosovilskiy et al., 2021) empowered the community to flip the paradigm. No- tably, ViLT (Kim et al., 2021) initializes with VIT, so it places visual representations as the initial concep- tual space to which language must adhere. Addition- ally, there are engineering benefits to this paradigm as it removes the computationally expensive need
ViLT (supervised with ImageNet) Ours (no BBox/class supervision)
Figure Aligned patches with the word thrown. Our model (VLC) produces a more meaningful distri- bution over the patches. We argue for the following factors towards learning such faithfully-alignedd and flexible vision-language representations: (1) Masked Image Modeling, (2) Avoiding a vision-only processing stack, (3) Avoiding priors from ImageNet pretraining.
Equal Contribution.

---

Published in Transactions on Machine Learning Research (09/2023)
for Region of Interest (ROI) extraction. However, because VIT is trained with supervised ImageNet labels, its representation may be constrained by the finite label space. The ImageNet concept space is still somewhat linguistic in nature when initialized, and requires expensive annotation, a hindrance to scaling to arbitrarily many visual categorics.
We take the important next step and remove the need for supervised pretraining. An unsupervised visual semantics is learned via Masked Auto-Encoder, (He et al., 2022) before language is integrated. This leads to both a better performing and more general model. In addition, every component can be improved and scaled with unsupervised and weakly aligned data removing the need for future annotation efforts while still scaling to open-vocabulary domains in the wild.
Our Vision-Language from Captions (VLC) model matches or outperforms nearly all vision-language transformers despite being 1. Smaller and faster at inference, 2. Avoiding use of ROIs, and 3. Not leveraging object-level supervised labels for pretraining. For downstream usage, token representations can be naturally adapted to producing class-labels and bounding-boxer with moderate finetuning. We demonstrate strong performance of VLC across a comprehensive set of benchmarks covering V+L understanding, retrieval, reasoning and grounding. Performance also contimues to improve with data and model size scaling, and as it relies only weak alignment of image-text pairs. Our ablation study shows that key to consistence improvement on ream lasks is the masked image modeling objective, which is in sharp contrast to existing findings. Finally, we provide several analyses on the underlying representations to understand what our models are learning and guide future V+I. transformer rescarch.
2 Related Work
Vision-Language Modeling. Based on how they encode images, most existing work vision-language modcling fall into three categories. The first category (Lu et al., 2019; Tan & Bansal, 2019; Li et al., 2019; Su et al., 2020; Chen et al., 2020b; Li et al., 2020; Zhang et al., 2021; Li et al., 2021b; Qi el al., 2020) focuses on using pre-trained object detectors to extract region-level visual features (e.g., by Faster R-CNN (Ren et al., 2015)). Extracting region-level features requires pretrained object detectors with high-resolution inputs that can be time-consuming. To tackle these two issues, the second category (Huang et al., 2020; 2021; Jia et al., 2021) proposes to encode images by using grid features from convolutional neural networks. The third category (Dou et al., 2022a; Xue et al., 2021; Li et al., 2021a; Radford et al., 2021) uses a Vision Transformer (ViT) (Dosovilskiy el al., 2021) as the image encoder and designs different objective functions for languae pretraining. Such approaches employ an encoder-decoder architecture coupled with image captioning as their pretraining tasks. Typically, PaLI (Chen ct al., 2023) and BLIP (Li ct al., 2022; 2023) adopt the pretrained ViT as their visual components. Flamingo (Alayrac et al., 2022) initializes its visual encoder from NFNet (Brock et al., 2021). These models not only require substantial computation already consumed by their initialization checkpoints, but have also oriented their focus towards the ultimate performance by relying on even larger multimodal training datasets. CoCA (Yu et al., 2022) and SimVLM (Wang et al., 2022c) are similar to our approach in that they do not initialize from well-trained unimodal components. But both works adopt the encoder-decoder architecture where the decoder only outputs text, which we believe cannot naturally handle grounding tasks. CLIPPO (Ischannen et al., 2023) unifies image and text representation by rendering text as images. While such a representation scheme unlocks the possibility of scene-text reading, its performances on GLUE, VQA and image-text retrieval still lag far behind models that encode text from tokens. The most similar work to ours is ViLT (Kim et al., 2021). ViLT does not use pretrained object detectors extra visual embedders for visual embedding, but still needs weights pretrained on ImageNet-211K for initialization. Another related study, VirTex (Desai & Johnson, 2021), aligns with a similar objective ours, which is to train vision-language representations with reduced data annotation requirements. Differing from prior research, we offer a re-examination of the roles played by (a) unimodal initialization, (b) unimodal architectural components and (c) annotated pretraining data. With careful controls on model and data sizes, our results suggest that none of the above factors is absolutely important in achieving versatile lasion- representations. Additionally, our model exhibits superior performance when naturally extending to reasoning and grounding tasks that necessitate distinguishing part-whole relationships.
2

---

Published in Transactions on Machine Learning Research (09/2023)
Masked Language Modeling. Masked language modeling (MLM) and its auto-regressive counterparts are widely used in natural language processing for learning text representations. MLM (Devlin et al., 2019; Tay et al., 2022) trains a model to predict a random sample of imput tokens that have been masked in a multi-class setting. In vision-language pretraining, MLM has shown useful to enforee the consistency across modalities (Dou et al., 2022a; Zhang el al., 2021; Li et al., 2020; Kim et al., 2021). In vision-languuae modeling, we randomly mask some of the input tokens, and the model is trained to reconstruct the original tokens given the masked tokens and their corresponding visual inputs. To be consistent with previous work, we follow the default settings of training BERT (Devlin et al., 2019) for masked language modeling.
Masked Image Modeling. Masked image modeling (MIM) is pretext task to learn representations from images corrupted by masking, Inspired by the success of masked language modeling (MLM) in NLP, different masked prodiction objectives have been proposed for image tasks. iGPT (Chen et al., 2020a) predicts unknown pixels of a sequence. ViT (Dosovitskiy et al., 2021) predicts mean colors of masked patches. BEiT (Bao et al., 2022) proposes to use a pre-trained discrete variational autoencoder (dVAE) (Ramesh et al., 2021) to encode masked patches. MaskFFatt (Wei et al., 2021) predicts HoG (Dalal & Triggs, 2005) features of the masked image regions. SimMIM (Xie et al., 2022) and MAE (He et al., 2022) predict RGB values of raw pixels by direct regression. MTM has also been explored in the field of vision-language representation learning by either regressing the masked feature values (Tan & Bansal, 2019; Chen et al., 2020b; Dou al., 2022a; Kim et al., 2021) or predicting a distribution over semantic classes for the corresponding image region (Chen et al., 2020b; Lu et al., 2019; Su et al., 2020). In contrast to previous approaches (Chen et al., 2020b; Kim et al., 2021; Dou et al., 2022a) that show MIM does not contribute to or hurt the performance on downstream tasks, we show that using MIM can consistently improve the performance as the training steps increase.
2.1 Model Architecture
Our aim is a vision-language transformer that can be trained without the need for expensive object-level supervised labels (e.g., class labels or object bounding boxes). More concretely, our results empirically show that such object based supervised signals are not necessary for vision-language protraining. To this end, we use a ViT-based framework to learn multi-modal representations by 1) intra-modal reconstruction through masked image/lamn /language modeling; 2) inter-modal aligmment through image-text matching. The architecture of our proposed VLC framework is illustrated in Figure 2. VLC consists of a modality-specifie projection module 3.1, a multi-modal encoder 3.2 and three task-specific decoders 3.3. We aim for minimal visual and textual embedding designs during pretraining.
3 Method
3.1
Sppelit Projection Module
While most of existing methods rely on complex ResNeXt (Huang et al., 2020) or object detection com- ponents (Chen et al., 2020b; Lu et al., 2019; Zhang et al., 2021; Li et al., 2020), we use a trainable linear projection layer to map flattened visual patches to the visual embedding space. The patch embeddings are represented as v = (ui...., "x} E xx where K is the number of image patches and is the hidden dimension of our model. For text embedder, sentence into WordPieces (Wu et al., 2016). We then adopt word embedding lookup layer to project tokenized words to the textual embedding space. Here we use w = [WELS. WI,..., wr} E Txd to represent the token embeddings, where T is the number of tokens and the special token CLS denotes the start of the token sequence. We encode patch and token positions scparately by yoo E R xl and woss E Ixddd We use plyee E 1 and tope E IRd as modality-type embeddings to distinguish the modality difference between patch and token embeddings. The final representations of each patch v, and token wy are calculated as
follow BERT (Devlin et al., 2019) to tokenize the input
i =LyeerNormm;; + upo + otup), and
(1) (2)
G Loyer + whee + wpee).
3

---

Published in Transactions on Machine Learning Research (09/2023)
Patch masks
Token position embedding
MASKK
posttion embeddlngg
[CLS]
Mocality-spcfie embedding
The yellow and blue bird standing on a branch
The
embedding
ITM head
Match or not
[MASK]
Multi-layer Transformer
MLM head
branch
MIM head
Linear Projection
sppecity--pecific
Multimodal Encoder
Task-specific
Projection
Decoder
Figure The overall architecture of VLC. Our model consists three modules: (1) Modality-specific projection. A simple linear layer projects image patches and token embeddings into the same representation space; (2) Multi-modal encoder. We use 12-layer VIT (Dosovitskly et al., 2021) initialized from MAE (He et al., 2022) (ImageNet-1K, without labels) pretrain multi-mode representations with masked mmage/languag, modeling (MIM/MLM) and image-text matching objectives. This work justifies the importance of the MIM objective throughout V+I. pretraining in addition to providing good initializations for the Transformer encoder.
Transformer backbone; (3) Task-specific decoder. We
3.2 Multi-modal Encoder
To learn the contextual representations from both visual and textual modality, we follow single-stream approaches (Kim al., 2021; Chen et al., 2020b) and use the ViT-B/16 architecture as our multi-modal encoder. ViT-B/16 consists 12 alternating layers of multiheaded self-attention (MSA) and MLP blocks. LayerNorm comes before every block and residual connections after every block (Dosovitskiy et al., 2021). We use merged -tteetinn (Dou et al., 2022a) mechanism to fuse the visual and textual modalities. More specifically, we concatenate the token and patch embeddings together as {WCLS, W1,..., 0r, 61...., tx then feed them into the transformer. We use the hidden states h at the output of the last layer of the encoder as the contextual representations (heLS ; ,i.,..,,, hy,,.. hg) In sharp contrast to existing approaches that
use object detectors, visual detectors pretrained with supervised labels or pretrained language models (e.g., BERT, Roberta), we initialize our model with MAE pretrained on ImageNet-1K with no labels.
3.3 Pretraining Objectives
To learn a universal visual and textual representation for rision-and-languael tasks, we apply self-supervised methods to pre-train model large aggregated dataset. Unlike previous approaches that only mask text tokens, we randomly mask both image palches and text tokens simultaneously. We train our model with masked image modeling (MIM), masked language modeling (MLM), and image-text matching (TTM).
Masked Language Modeling. In language pretraining, MLM randomly masks input tokens, and the model trained reconstruct the original tokens based on unmasked context. Following BERT (Devlin et al., 2019), we randomly mask text tokens with a probability 0.15, and replace the masked ones Wm with special token [MASK]. The goal is to predict the masked tokens based both non-masked text takens Wim
4

---

Published in Transactions on Machine Learning Research (09/2023)
and image patches Mm- The learning target LMLM can be formulated as
LMLM (w log log (mWm//// Wim V(m)
(3)
We use a linear layer with default parameters (Devlin et al., 2019) as the MLM head to output logits over the vocabulary, which are used to compute the negative log likelihood loss for the masked text tokens.
Masked Image Modcling. Existing approaches explore MIM either by regressing the masked features values (Chen et al., 2020b; Kim et al., 2021; Xue et al., 2021) or by predicting a distribution over semantic classes for image region (Chen et al., 2020b; Lu et al., 2019; Dou et al,, 2022a). In contrast, we randomly mask image patches with a probability of 0.6, and reconstruct the missing pixels based on both non-masked tokens Wim and patches v (as in MAE (He et al., 2022)). The learning target LMIM can be formulated as
CMIM B( (fm.:) Vim),
(4)
where the feature regression objective f is to regress the masked image patch representations to pixel values. We use &layer transformer as the MIM head For masked image patch Vt, the objective f can be formulated as: [us/[Wmm: V ||r(h²) wi|l. Each output of the MIM head is vector of pixels representing a patch. Different from the observations in ViLT (Kim et al., 2021) and METER (Dou et al., 2022a), we find that MIM consistently improves the performance on downstream tasks as training progressos.
Image-Tess Text Matching. Given a batch of image and text pairs, the ITM head identifies if the sampled pair is aligned. We randomly replace the aligned image with a different one with a probability of 0.5. We use the special token [CLS] as the fused representation of both modalities, and feed hccs to the ITM head. The learning target LITM can be formulated as
CITM = (w logply/w, v),
(5)
Where y E {0, 13 indicates whether the image and text are matched (y = 1) or not (y = 0). We use a single linear layer as the ITM head and compute negative log likelihood loss as our 1TM loss. We weight the pretraining objectives equally so the full pro-training objective is:
C = LMLM + CITM + CMIM
(6)
For a fair comparison with existing approaches, We do not include image-text contrastive loss (Li et al., 2021a; 2022), momentum distillation (Li et al., 2021a) and other techniques in our implementation.
4 Experimental Setup
4.1 Pre-training Datasets
Following previous work (Chen et al., 2020b; Kim et al., 2021; Li et al., 2021a; Dou et al., 2022a), our pre-training corpus comprises four commonly used vision-language datasets including COCO (Lin et al., 2014), Visual Genome (Krishna et al., 2017), Google Conceptual Captions (Sharma et al., 2018) and SBU Captions (Ordonez et al., 2011), totalling 4.0M unique images and 5.1M image-text pairs. To show the benefits of data-scaling, we also use the VinVL (Zhang et al., 2021) pretraining data which includes Flickr30k (Young et al,, 2014), GQA (Hudson & Manning, 2019), VQA (Goyal et al., 2017), VG-QAs (Krishna et al., 2017) and subset of Openlmages (Krasin et al,, 2016). This larger pre-training corpus contains 5.65M unique images (sce detailed statistics in Appendix A). We carcfully control the amount of protraining data used for both the initialization checkpoint (e.g., ImageNet. (Russakovsky el al., 2015), MSCOCO Lin et al. (2014), WebLI Chen et al. (2023)) and for dision-langual pretraining. Note that many recent works tend to underestimate the extent of data and computational resources necessary for their unimodal initializations. Consequently, this oversight inadvertently obscures the fact that their final outcomes are heavily reliant on considerably more resources than would constitute fair comparison. Additional analysis can be found in Appendix B.
5

---

Published in Transactions on Machine Learning Research (09/2023)
4.1.1 Fine-tuning Datasets
We fine-tune our model on image-text retrieval: Flickr30K (Plummer et al., 2015) and MSCOCO (Lin et al., 2014), image-text understanding: VQAv2 (Goyal et al., 2017) and NLVR (Suhr et al., 2018), and imago-text grounding: Refcoco (Yu et al., 2016; Mao et al., 2016) tasks. We fine-tune all encoder layers together with task-specific heads. For retrieval tasks, we follow the standard splits and evaluate our models in the finetuning settings. For VQAv2, we follow the standard practice (Chen et al., 2020b; Li et al., 2021a) to train the models with both training, validation and additional question-answer pairs from Visual Genome while reserving 1, 000 validation samples for internal validation. For Refcoco, we propose a new algorithm to obtain bounding box outputs based on affinities between patch and text encodings. This only requires finettunin the encoder backbone without introducing extra parameters. Finally, ablation studies validate our key design components. More experimental dctails can be found in Appendix C and D.
4.2 Implementation Details
We pretrain two variants of the multi-modal encoder which uses a 86M parameter ViT-B/16 denoted VLCBase and 307M parameter VIT-I,/16 denoted as VLC Both variants are initialized with MAE pre-trained on ImageNet-IKK without labels. For text imputs, we tokenize text with the bert-base-uncased and bert-large-uncaseed tokenizer, respectively. The text embedding parameters are learned from scratch, in lieu of loading pre-trained BERT weights. We randomly mask image patches with a probability of 0.6 and text tokens with a probability 0.15. To accelcrate training, we follow MAE (He et al., 2022) and skip the mask token [MASK] in the encoder and only apply it in the lightweight decoder. We use AdamW (Loshchilov & Hutter, 2018) with weight decay of 0.01. The learning rate is warmed-up to le in the first 10% of total training steps and is decayed to zero for the rest of the training following a linear schedule. During pre-training, we resize the shorter edge of input images to 384, take random image crops of resolution 384 x 384, and apply
(Cubuk et al., 2020). We pre-train for 200k steps with a batch size of 4, 096. For the parameter estimation, we exclude the textual embedder as it is shared by all vision-languaae transformers. We also exclude the parameters of all the auxiliary heads as they are only required during protraining. Unless otherwise specified, we use the base version of VLC for downstream evaluation and visualization.
5 Downstream Experiments & Results
We evaluate the effectiveness of our pre-training strategy under both zero-shot and fine-tuning settings. VLC is competitive across a diverse set of standard V+L benchmarks, broadly categorized into Image-Tex Text Retrieval, Image-Text Understanding, and Image-Tex: Grounding. In addition fine-tuning based evaluations, we also perform inference-only experiments on multiple challenging benchmarks that test reasoning (Thrush et al., 2022; Ji et al., 2022) and open-domain question answering (Schwenk et al., 2022).
5.1 Zero-shot Experiments
First we verify the strength of our approach in a zero-shot setting before investing in larger fine-tuning results. We adapt our model to perform zero-shot visual reasoning on the Kilogram (Ji et al., 2022) dataset. Kilogram challenges a model to recognize an abstract Tan- gram shape from a language description. Tangram shapes drastically differ from natural scenes as they only contain abstract and implicit visual clues. Hence, recognition demands more sophisticated reason- ing about the interplay between visual and linguistic cues. The task is formulated into a 10-way classification problem where the model chooses the most relevant image given an abstract shape description. Specifically, we rank image relevance by the logits calculated by the pretrained imagc-text matching head. Table 1 shows that our model consistently outperforms ViLT across all input conditions. This sup- ports our design consideration that freeing up the model from priors of limited ImageNet concepts helps it.
Input Condition
ViLT Ours
Human
WTHOLE+BLACE
12.9
13.8
47.7
PARTS+BLACK
12.5
15.2
49.1
WHOLE+COLLO
11.7
13.9
19.5
PARTS+COLOR 10.7
13.5
63.0
Table Zero-shot accuracy on Kilo- gram (dev), a challenging image-text matching task for abstract visual recog- nition. VLC consistently outperforms ViLT across all input conditions.
6

---

Published in Transactions on Machine Learning Research (09/2023)
Text Retrieval
Image Retrieval
Model
Flickr30K
(1K)
MSCOCO (5K)
Flickr30K (1K)
MSCOCO (5K)
Params
@1
@5
@10
@1
@5
910
@1
@
@10
81
(55
@10
ALBEF (Li et al., 2021a)
163M
94.3
99.4
99.8
73.1
91.4
96.0
82.8
96.7
98.4
56.8
81.5
89.2
VinVLiaege (Zhang ct al., 2021)
452M
75.4
92.9
96.2
58.8
83.5
90.3
UNeTERLage (Chen et al., 2020b)
371M
87.3
98.0
99.2
65.7
88.6
93.8
75.6
94.1
96.8
52.9
79.9
88.0
METER--Winpaec (Li et al., 2020)
288M
92.4
99.0
99.5
76.2
93.2
96.8
79.0
95.6
98.0
54.9
81.4
89.3
PixeIBERT (Huang et al., 2020)
144M
87.0
98.9
99.5
63.6
87.5
93.6
71.5
92.1
95.8
50.1
77.6
86.2
VILT (Kim et al., 2021)
86M
83.5
96.7
98.6
61.5
86.3
92.7
64.4
88.7
93.8
42.7
72.9
83.1
VLCBuse (with 5.6M examples) VLCLage (with 5.6M examples)
86M
89.2
99.2
99.8
71.3
91.2
95.8
72.
93.4
96.5
50.7
78.9
88.0
307M
94.4
99.6 99.9
76.7
94.5
97.3
79.1
95.8
98.2
58.4
84.0
91.1
Table 2: On Image-text retrieval tasks, we see substantial gains over state-offthe-art methods with bounding- box pretraining. 'ALBEF uses specifically designed coarse-to-fine objectives for the image-text retrieval task.
generalize to arbitrarily many linguisticalll-sseeifee visual categories. Further improvement may come from additional finetuning to familarize our model with the Tangram shape domain, which is left for future work.
5.2
Finetuning Experiments
Next we investigate the role that fine-tuning has on our downstream performance across traditional retrieval and understanding benchmarks, before analyzing training and performance curves.
5.2.1 Image- Text Retrieval
We begin with a proof of concept experiment, evaluating our model on the Karpathy splits of the Flickr30K (Plummer et al., 2015) and MSCOCO (Lin et al., 2014) benchmarks. Table 2 compares VLC strong multimodal transformers which leverage ROIs, more parameters, and are pretrained on ImageNet classification. Note that as most of detection- based models have the advantage of using Faster R-CNN (Ren et al., 2015) pre-trained on VG (Krishna et al., 2017) MSCOCO (Lin et al., 2014). The closest comparison to VLCBase is ViLT (same model size), though ViLT still requires more supervised data in the form of ImageNet pretraining for VIT (Dosovitskiy et al., 2021). When comparing to dual-encoder models, our VLCCarge achieves competitive results across all settings. ALBEF uses pre-trained ViT and BERT model for initialization. Additionally, it specifically designs the coarsc-to-finc objectives while we directly finc-tunc the pre-trained ITM head for retrieval lasks. Thus we treat ALBEF as a strongest available baseline.
5.2.2 Image- Text Understanding
Table 3 presents results on two popular image-text understanding datascts: VQAv2 and NLVR². We use the same training data as ViLT denoted as 4M and VinVL denoted as 5.6M.
Comparison to models supervised /oaddd/iiitiiiiii iniite with ImageNet bounded boxes. Most of these models use object detectors pretrained on VG (Krishna et al., 2017) or MSCOCO (Lin et al., 2014) to extract region features. Object detectors help VQA tasks as they mainly ask about objects. Within the similar scale of pretraining data, our model achieves competitive performance on both tasks. Note that our model uses 384x384 or 576x576 as input resolution during our fine-tuning stages. This resolution is much lower compared with previous work using x00 1333 (Lu et al., 2019; Chen et al., 2020b). In particular, VLn (Zhang et al., 2021) has a multi-stage pre-training for its object detector that has access to ImagcNet-5K (Xie et al., 2017) (6.8M images from 5K classes) and four object detection datasets (Shao et al., 2019; Krishna et al., 2017; Lin et al., 2014; Krasin et al., 2016) (2.5M images with bounding box annotations). Our VLCLarges which has a similar model size, achieves better performance on the VQA task and competitive results on the NLVR task without any supervised initialization.
VILT uses V/T-B/32 pretrained with ImageNet.211 and finetuned on ImageNet- with supervised labels.
7

---

Published in Transactions on Machine Learning Research (09/2023)
VQAv2
test
NLVR
Model
Param
-dev
-tt
dev
test
Supervised ImageNet Bounded Bores
VILBERT (Lu et al., 2019)
274M
70.55
70.92
LXMERT (Tan & Bansal, 2019)
240M
72.42
72.54
74.90
74.50
VisualBERT (Li et al., 2019)
170M
70.80
71.00
67.4
67.0
UNITER (Chen al., 2020b)
371M
73.82
74.02
79.12
79.98
OSCARLarge (Li et al., 2020)
371M
73.61
73.82
79.12
80.37
VinVLa (Zhang et al., 2021) (5.6M)
452M
76.52
76.60
82.67
83.98
Supervised ImageNet Classes
METER-Swingase 1 (Dou et al., 2022b)
288M
76.43
76.42
82.23
82.47
ALBEF (Li et al., 2021a)
163M
74.54
74.70
80.24
80.50
Visual Parsing (Xue al., 2021)
180M
74.00
74.17
77.61
78.05
PixelBERT (Huang et al., 2020)
14M
74.45
74.55
76.5
77.2
ViLT (Kim et al., 2021)
86M
71.26
75.70
76.13
No supervised classes or bounding bozes
VLCBase (ours 4M)
86M
72.98
73.03
77.04
78.51
VLCBase (ours 5.6M)
86M
74.02
74.0
77.70
79.04
VLCLarge (ours- 5.6M)
307M
76.95
77.02
82.27
83.52
Pre-trained initialized with > 10M data
X-VLM (Zeng et al., 2022) (16M)
216M
78.22
78.37
84.41
84.76
BLIP (Li et al., 2022) (129M)
252M
78.25
78.32
82.15
82.24
OFA (Wang et al., 2022b) (54M)
930M
82.0
82.0
CoCa (Yu et al., 2022) (L.8B)
2.1B
82.3
82.3
86.1
87.0
Table 3: Despite that VLC is only pretrained with alakly-aligned image-caption pairs, it matches outperforms larger and more hcavily supervised approaches within a similar scale of training data and model size. VinVL uses the object detector trained with 6.8M labeled imageNet. images and 2.5M images with bounding box annotations. *METER-Swingee uses Swin-B trained with 14M labeled ImageNet as the image encoder and pretrained Roberta as the text encoder.
Comparison to models with supervised ImageNet classes. Most of these approaches use additional visual embedders together with pretrained BERT as their backbones. For example, ALBEF (Li et al., 2021a), Visual Parsing (Xuc et al., 2021), PixcIBERT (Huang et al., 2020) use pre-trained ViT-B/16, Swin transformer, ResNeXt-152 as their visual embedder, respectively. All these embedders are trained with HETER-SwinBan uses Swin-B/16 pretrained with more than 14M
labeled ImageNet data. In particular, labeled ImageNet22K images as the image encoder and pretrained Roberta as the text encoder. Experiments show that our model achieves better results than larger and more heavily supervised approaches.
Note that some models have access to much larger scale of data during pretraining, For example, METER- CLIP-ViTH Base uses CLIP the image encoder which is trained with 400M image-text pairs. X-VLM (Zeng et al., 2022) uses the same image encoder as METER-Swingae but with extra bounding box annotations and two times the training data. These models achieve higher performance, but comparisons are out of scope.
5.2.3 Image- Text Grounding
Since the model was not pretrained to generate bounding boxes, we propose a novel algorithm which produces bounding box based on patch-text affinities in a single forward pass. This results much less inference time and higher parameter efficiency compared with competitive pipelines tailored to bounding box generation. See Figure (Left) for overview of the proposed grounding workflow and Appendix G for details.
8

---

Published in Transactions on Machine Learning Research (09/202))
Affinity map
p1x1
tast lan Ph Moathhararttaiio
Redio to
Supardised Fite duning
Predicted patch-text affinity map
Werus stno Armen Aendaa woweoca Aommana
Predicted mask GTH
pakiit Jos No
Tokan raprose
Projectio Word embe
Projection
Phodss
CLR
Inforence
2nd where irput the selected region. The mask is projected back to the original scale.
Teget siikaaly maa
1st run
Figure 3: Left: Extending VLC to efficiently performing image-text grounding tasks: We finetune the Transformer backbone with supervision on similarities between patch and text representations. During inference, a bbox/mast can be obtained from the affinity scores with minimal computation overheads. Right: Reasonable segmentation masks resulted from interpolating patch-text affinities. Running the same inference procedure twice with coarso-to-fine resolution improves procision.
We finetune the encoder on the training sets of Refcoco, Refcoco+, and Refcocog (Yu al., 2016; Mao et al., 2016) following the umd split. We convert ground-truth bounding boxes to patch-text affinity scores and use them to finetune the preceding representations. At inference time, we algorithmically predict bounding box for each referential expression from affinity scores between the last layer patch and text representations.
Comparison to modularized models Previous models either an Rol Reegion-of-Tnterest) extractor to produce candidate bboxes to choose from (Chen et al., 2020b; Gan et al., 2020; Cho et al., 2021; Subramanian et al., 2022), directly predict the coordinates and dimensions of bboxes in the image coordinate system (Deng et al., 2021). The former approach relies on an off-the-shelf detector, while the latter requires gating mechanisms to infuse linguistic information into a visual backbone. Our approach outperforms all previous approaches with modular designs (Table 4, Top), meanwhile achieving the best inference-lime efficiency. This justifies our unified approach in which both understanding and localization tasks can benefit from large-scale representation learning.
Comparison to unified models that have far more parameters or computation Most of performant models under comparison incorporated bounding box annotations from VG (Krishna et al., 2017) in their pretraining data (Table 4, Bottom). With a much lighter architecture and much less annotated data, our model already achieves competitive performance. This holds promise that the performance will continue to improve resolution and data are scaled up during pretraining.
Generalization to dense prediction Moreover, demonstrates that interpolating well-aligned path- text affinities naturally results in segmentation masks. Table 6 reports pixelwise-lol IoU between our interpo- lated masks and ground-truth masks on Refcoco/+/ We have doubled the performance of TSEG (Strudel et al., 2022), the previous state-of-the-an method on Referring Expression Segmentation without training
Model
Refcoco/+/giie ((tcoco////((ud)) val
TSEG-CRF (Strudel et al., 2022)
25.95
22.62
23.41
VLCBae (ours)
50.65
45.88 46.37
Table We perform Referring Expression Segmenta- tion through interpolating low-resolution affinity maps into segmentation masks without dense supervision. on ground-truti masks. Note, performing second forward pass with zoomed-in regions leads to significantly finer mask contours (Figure 3 Right). We leave closer investigation of this inference trick to future work.

---

Published in Transactions on Machine Learning Research (09/2023)
Top-
Model
Time
Param Refcoco/ //op//////mmdddd val
Model
Base Large
VL-T5 (Cho et al., 2021)
5.9x
290M
71.2
2e Suseerriee Supervieed
VIT-B/16 (Dosovitakly et al., 2021)
77.9
76.5
TransVG (Deng et al., 2021)
2.8x
169M
80.83
68.00
68.71
DeiT
(Touvron et al. 2021)
83.1
Modular
TransVG + (Deng et al,, 2022)
86.28
75.39
76.18
Swins.. (Liu et al., 2021)
84.5
QRNet (Ye et al., 2022)
5.5x
273M
84.01
72.94
73.03
DINO (Caron et al., 2021)
82.8
RefTrans (Li & Sigal, 2021)
85.65
77.55
79.25
MoCo v3 (Chen al, 2021)
53.2 81.1
SeqTR (Zhu et al., 2022)
44x
108M
83.72
71.45
74.86
MaskFeat (Wei et al, 2021)
83.6 85.7
UNITERMang (Chen et al., 2020b)
371M
81.41
75.90
74.86
SimMIM (Xio et al., 2022)
83.8 85.4
ViLLAtarge (Gan et al,, 2020)
371M
82.39
76.17
76.18
BETT (Bao et al,, 2022)
84.6 85.2
VLCHase (ours)
1.0x
110M
86.67
77.44
80.43
MAE (He et al., 2022)
83.6 85.9
VLC (ours)
84.5 86.3
OFAMediom (Wang et al., 2022a)
98M
85.34
76.09
78.76
Mdetz-R101 (Kamath et al., 2021)
3.0x
185M
86.75
79.52
81.64
Table 5: Our pretraining does not compro- mise the quality of image representations, as evidenced by our superior results on Im- ageNet classification. Supervised models are pretrained on ImageNet 1K and self- supervised models are evaluated by end- to-end fino-tuning. *BEiT uses a DALLE pretrained tokenizer.
Unifed
Mdetr-ENB3 (Kamath et al., 2021)
2.6x
153M
87.51
81.13
83.35
UniTAB (Yang ot al., 2022)
19.4x
198M
88.50
80.97
84.58
OFAgase (Wang et al., 2022a)
6.0x
180M
88.48
81.39
82.29
OFAtarge (Wang ot al., 2022a)
11.5x
470M
90.05
85.80
85.89
OFA (Wang et al,, 2022a)
930M
92.04 87.86
88.07
Table 4: Referring Exprcssion Comprchension organized by: efficency--erformmnce and -versatility. Modularized baselines include bespoke components in their architecture. Unified base- lines emphasize versatility over efficiency, and are pretrained with bounding box annotations. Our model is much lighter and faster. Params counts all parameters required for inference, including Rol-proposals, the architecture, and output heads (sce SG).
Better Grounding translates to compositional reasoning We find that the finctuned patch-
text affinities translate to greater compositional reasoning performance. We report inference-
only results on Winoground (Thrush et al., 2022) with our Refcoco-finetunee model in Table 7.
Winoground requires pairing up two sets of images and sentences with minimally contrastive semantics. Instead of directly predicting a pairing logit. using the image-text matching head, it is more effective to measure image-sentence association via grounding success conditioned on an input sentence. Concretely, by treating the max affinity as the image-sentence. matching score, our
Model
Text Image
Group
MTurk Human
89.50
88.50
85.50
Random Chance
25.00
25.00
16.67
UNITER
32.25
13.25
10.00
VILLA Buse
30.00
12.00
8.00
model achieves a state-of- seate-of-the-art Group Score and outstanding Image Scores on Winoground. This superior reasoning performance again verifies that our pretrained representations are more capable of using text//mages to disambiguate each other.
ViLT
34.75
14.00
9.25
CLIP Base
30.75
10.50
8,00
FTAVA-ITM
32.25
20.50
14.25
VLCBaae
28.00
19.75
12.50
UNITER
38.00
14.00
10.50
VILLALarge
37.00
13.25
11.00
5.3 Ablations
VL.Giagg
32.00
20.00
14.75
Pretraining Objective We ablate different combinations of objectives and train VLCBase with 4M image-text pairs. Figure 4 (left) shows that, as the training steps increase, there is a consistent improvement for VLC with MIM. This contrasts to findings in previous work (Dou et al., 2022a; Kim et al., 2021; Li et al., 2021a). We compare VLC with different initializations and the effects of unimodal pretraining in Appendix E.
Table 7: Winoground is a chal- lenging test-only set for visio- linguistic compositional reason- ing. VLClarge competitive among similar-sized models with- out second-stage pretraining.
Downstream Adaptation Since we finetune for 30 epochs on the grounding task vs. 10 epochs on the retrieval or understanding lasks, we would like to understand whether the grounding capability is simply a result of intensive finetuning. To this end, we experiment with our VLC checkpoint as well as other three pretrained checkpoints: ViT (Dosovitskiy et al., 2021), ViLT (Kim et al., 2021) and MAE (He et al., 2022).
Please refer the Winoground (Thrush et al., 2022) paper for how their evaluation metrics are defined.
10

---

Published in Transactions on Machine Learning Research (09/2023)
VLC-full
0.8
VLC w/o MIM
VLC w/o ITM
72.08
VLC W/o MLM
0.7
VOAV2 accuracy (%)
72
71.69
71.8
Refcocog val performance
0.6
70.75
1.15
0.5
09.52
89.06
0.4
VLC Large
68
67.06
0.3
MAE_larg
VLC
66.55
65.83
0.2
MAE
95.45
VILT
65.45
0.1
25
50
75
100
125
150
175
200
10
15
20
25
30
35
Number training steps (K)
Number finetuning epochs
Figure 4: Left: Role of objective functions. Our experiments show that VLCBase sees gains from including MIM and that increased training steps continues to improve VQAv2 performance. Right: Comparing the effects of different pretrained V-L checkpoints on downstream adaptation. Our experiments show that the pretrained VLC representations enable faster finetuning and better downstream performance.
We follow the same finctuning and inference procedures for all chockpoints. Figure (right) plots validation accuracy curves against the number of fineluning epochs. VLC exhibits considerable advantage right from the beginning, and maintains clear margin until convergence. The relative advantage of VLC becomes even greater when compare the Larye model variants. This result validates that our proposed pretraining scheme provides better foundation for finetuning, easing the computation burden of downstream adaptation.
InitializationnnScllnn In Figure 5, we compare VQAv2 lest-dev accuracy between VLC with three baselines that use pretrained object detectors and BERT for initialization. We see that while supervised initialization provides strong priors for the VQA task (e.g., VinVL), scaling the model size only has marginal improvement. As a comparison, our model is initialized with MAE pretrained on ImageNetIK without labels. There is a substantial gain when scaling to a larger model.
24
6 Understanding the models
UNITER
OSCAR
Figure 5: Comparison of the benefit of model scaling. While models initialized with supervised checkpoints inherit strong priors, VLC cxhibits the most substantial improve- ment when scaling the model size.
VLC is designed in importantly different ways from traditional bounding-box based approaches. In particular, while the visual stack traditionally frozen those models, now the entire "backbone" is learnable. Also, where previously, the goal was to "map" vision to language, the two learned jointly. We therefore take this opportunity to investigate the models to better understand how their behaviors differ due to the two key design choices. For fair comparison with ViL.T, we use VLC which is trained with the same model architecture and image-text pairs.
Understanding Patches. We begin with a simple patch clustering visualization (Figure 6). Without the inclusion any language, we can simply cluster (and color) the visual patch emboddings of ViLT and VLC. ViLT relies on on larger patches (32x32) for higher resolution (384x640) We instead use smaller patches and lower resolution (16x 16 for 384 384). It is easy to how both models identifying key semantic
11

---

Published in Transactions on Machine Learning Research (09/2023)
Original Image
ViLT clusters
VLC clusters
Figure 6: Visualization of patch clusters for example image produced from ViLT (many densely clustered patches) versus VLC's more fine-grained and dilfuuse represenlations. We believe this represenlational difference makes for easier and faster learning and scaling akin to "fast mapping" in language acquisition.
regions of the image (c.g. the rug, painting and plant). Also note, both models incorrectly place the painting and plant in the same cluster.
We leverage the nocaps dataset (Agrawal et al., 2019) to investigate representation diversity at scale. Nocaps provides captions for images from three buckets: inside COCO (in-domain), similar to COCO (near-domain), and outside COCO (out-domain). We compute representation similarities between patches and nouns for these three image buckets separately, and soek to answer the following questions: (1) Are VILT patches more tightly clustered perhaps due to the discriminative training objective, and (2) How do both models' representation diversity change for images more (or less) like ImageNet images. In Figure 7,
several trends. First, ViLT's "most similar" patch to the noun rarely has a similarity greater than 0.1, indicating that information initially coming from two modalities are not well integrated into cach other's representations. Second, we see the mass of patch-noun similarity distributions for ViLT slightly shift towards zero as we move from in-domain to out-domain, indicating that ViLT's difficulty in finding alignments to novel words. VLC has a significantly smoother distribution of patch-noun similarities. VLC often represents a nontrivial set of patches largely close to nouns across all image buckets. Also, the centers of similarity mass do not shift much those in ViLT plots we move to out-domain images. A caveat is that these plots do not justify the semantic meaningfulness of the alignment, but they do show starkly different behaviors between ViLT and VLC. Our findings are supported by more visualized examples in F.
Indomain Nour Probabily, Densty
reardomals Probabilty Denstty
outdomain Probabiity
17.5
20.5-
D03
35.0
15.0
15.3
12.5
X2.3
25-
5.0
25
2.5
60
60
0.4
2.5
03
3.4
65
08
53
62
Intoypatcl Seniientty Stores
1etersstl potul Siniaaraa boonss
Talon pulct Sieslerky Suxce
Figure 7: We plot the distribution of patch-noun similarities to compare the representation quality ween ViLT and VLCBase- ViLT rarely represents patches in great proximity to nouns, likely due to its discriminative pretraining objective. ViIT also tend do produce weaker aligument for images dissimilar ImageNet images. In contrast, VLCBase produces patch representations grongly-aligned corresponding nouns and does not show preference towards ImageNet images.
Image Classification. Given that the underlying visual representations are shifting through the cross- modal training, we run a simple image classification experiment to see the effects language training has on the underlying visual "backbone'. We compare VLC with stato-of-the-art models on ImageNet- 1K classification and report top-1 validation accuracy of a single 384 384 crop.
12

---

Published in Transactions on Machine Learning Research (09/2023)
The bird is on the branch with leaves alone
bird
branch
leaves
Figure 8: Finer-grained patch-token aliguments is implicitly learned from weakly-aligned pretraining data. We visualize patch-similarities with different tokens from the same caption to how our model uniquely represents them. It is interesting to note that our model attempts to differentiate branches from leaves.
As shown in Table 5, VLC learns generic representations which transferable to ImageNet classification. After finetuning ImageNet- IK, our model matches the performance of SwinBase (Liu et al., 2021) that is trained with supervised labels. Note that BEiT (Bao et al., 2022) is two-stage pretrained model of which the image tokenizer is trained on 250M examples of DALLE (Ramesh et al., 2021) data. Compared with MAE (He et al., 2022), our model learns competitive multi-modal representations from vision-language pretraining while retains high-quality image representations.
Evaluation on Open-domain VQA. To inves- tigate if the alignments between image patches and text tokens are semantically meaningful, eval- wate our VLC A-OKVQA dataset (Schwenk et al., 2022). A-OKVQA is more demanding than VQA (Goyal et al., 2017) for it requires commonsense reasoning about the scene depicted in the image. In multiple choice (MC) setting, model chooses answer from one of four options. In the direct answer (DA) setting, model can generate any text as its answer, which better applies to real-world scenar- ios. We use VLC as the mulitmodal encoder and pretrained BERT to generate answers. In Table 8, we compare VLC with large-scale pretrained dis- criminative models BERT (Devlin al., 2019) and ResNet (He et al., 2016), contrastive model CLIP (Radford et al., 2021), generative model GPT3 (Brown et al., 2020), and models specialized open-domain VQA. While CLIP stands out in the MC setting, performs worse than other base- lines in the DA setting, KRISP (Marino et al., 2021)
Model
Model Size MC-test DA-test
Raudom
25.36
Large-scule Pretrained model
BERT (Devlin et al., 2019)
110M
33.54
8.41
GPT-3 (Brown et al., 2020)
175B
35.21
11.49
ResNet (He et al., 2016)
23M
28.81
2.30
CLIP (Radford et al., 2021)
150M
51.01
7.10
Specialized model
ViLBERT (Lu et al,, 2019)
274M
41.5
25.9
LXMERT (Tan Bansal, 2019)
240M
41.6
25.9
KRISP (Marino et al., 2021)
300M
42.2
27.1
V VLC (ours)
196M
44.82
27.49
ClipCap (Mokady et al., 2021)
930M
51.43
25.00
GPV-2 (Kamath et al., 2022)
380M
53.7
40.7
Table 8: A-OKVQA under both Multiple-Choice (MC) and Direct Answer (DA) settings. *ClipCap uses pretrained CLIP+GPT2Lare. !GPV2 uses pretrained VinVI+T5maes and learns from Bing data. We report accuracy (%) returned from the evaluation server.
ensembles different pretrained image classification and object detection models to exact image features. a comparison, our VLC outperforms KRISP in both settings. This implies that our model provides richer and language-aware image features which conducive to open-vocabulary text generation. Note that ClipCap (Mokady et al., 2021) and GPV2 (Kamath et al., 2022) use either much more data for pretraining finetuning on the open-domain VQA task.
13

---

Published in Transactions on Machine Learning Research (09/2023)
7 Visualizations
Patch-based transformer architectures al- low for intuitive visualizations of the lexi- cal alignment. We first demonstrate that faithful alignment corresponding entity names or attributes already exist in the pre- trained representations. Note, there is explicit supervision fine-grained patch- token alignment, yet the model learns to do so from weakly-aligned data. In Figure 8, we show results from visualizing three different words the same caption for an image from COCO. Note that for the word branch, the model is actively temppting to
Caption with focus
Original Image
VILT
VLC
person on a beach holding kite string and a kite in the air
cat sitting on a chair, that is blue and yellow
avoid the abundant leaves. Second, since there is nothing about our model besides the MAE initialization that should be biased (as shown previously) towards ImageNet classes; we present three images in Figure 9 that highlight words not present in the standard ImageNet1K training split used by other models. Specifically, a noun (string), adjec- live (yellow), and verb (swinggngg) These demonstrate the general trend of ViLT often focusing on surprising locations.
baseball player swinging a baseball bat at a baseball
Moreover, Section 5.2.3 shows that finetun- ing can further incentivize the affinity pat- terns to behave like bounding boxes, which is the standard oulput format for localiza- tion. We visualize predicted alignment af- ter finetuning in Table. 9. We highlight a greater reasoning ability beyond recognition in terms of disambiguating visual entities basod on how they are referenced linguistically.
Figure 9: Out-of-domain concepts and images (i. not in ImageNet-1K). We visualize patch-similarittes with noun (top), an adjective (middle) and a verb (bottom), respectively. The model delicately avoids nearby but distinct concepts (e.g. the cat on the chair or irrelevant parts of the baseball field). More examples and analysis can be found in Appendix F.
8 Conclusion
We present Vision-Language from Captions (VLC), generic vision-language model pretrained with only image-caption pairs. It uses single linear layer project pixel stimuli and token embeddings into the same representation space, followed by Transformer blocks jointly modeling two modalities. By removing the dependency on image region proposals, our model is both (1) more data efficient, for it does not require pretraining-scale class labels or bounding box annotation, and (2) faster at inference, for it does not require a tedious vision-only branch.
Despite being lighter and faster, VLC performs competitively on diverse set of vision-language tasks, compared to existing approaches relying detection ImageNet supervision. The MIM pretraining objective encourages richer and language-aware visual representations, which implicitly results in finer-grained alignment between patch and token representations. With moderate downstream finetuning, VLC can be easily adapted to (1) perform multi-modal retrieval, (2) answer questions, (3) reason about visual information guided by free-form language, or (4) ground inguisticallly-eefeenii strong performance across nine downstream benchmarks clearly demonstrate the wide task applicability of unified vision- lasion-languae encoder. Our results appeal the community rethink the general multimodal pretraining pipeline and to reduce unnecessary cost of pretraining unimodal encoders. As performance of
objects into bounding boxes. VLC's
14

---

Published in Transactions on Machine Learning Research (09/2023)
Input image
Predicted affinity map
Predicted bbox reduced
Predicted & Ground-trute
the leftmost train
The model succeeded at distinguishing same same-type instances via absolute position.
the smallest train
The model succeeded at distinguishing same-type instances via absolute size.
Table 9:
Patch-token affinitics can be finetuned towards localizing objects specified by referring expressions.
VLC scales with increased training data, this opens an exciting avenue for large-scale weakly-supervised open-domain vision-language models.
References
Harsh Agrawal, Karan Desai, Yufci Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In ICCV, 2019. 12
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. NeurIPS, 2022. 2, 22
Hangbo Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. ICLR, 2022. 3, 10, 13
Andy Brock, Soham De, Samuel L Smith, and Karen Simonyan. piifommerforance large-scale image recognition without normalization. In International Conference on Machine Learning, pp. 1059-1071. PMLR, 2021. 2
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NewrIPS, 2020. 13
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging propertics in self-supervised vision transformers. In CVPR, 2021. 10
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In ICML, 2020a. 3
Xi Chen, Xiao Wang, Soravit Changpinyo, A.J Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustala, Lucas Beyer, et al. Pali: jointly-scaled multilingual language-image model. ICLR, 2023. 2, 5, 22
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers.
In ICCV, 2021. 10
Con-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal imago-text representation learning. ECCV, 2020b. 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 21, 23, 33
15

---

Published in Transactions on Machine Learning Research (09/2023)
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning, pp- 1931-1942. PMLRR 2021. 9, 10, 33
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V Le. Randaugment: Practical automated data augmentation with reduced search space. In CVPR Workshop, 2020. 6
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In CVPR, 2005,
Jiajun Deng, Zhengyuan Yang, Tianlang Chen, Wengang Zhou, and Houqiang Li. Transvg: End-to-end
visual grounding with transformers. In ICCV, 2021. 9, 10, 33
Jiajun Deng, Zhengyuan Yang, Daqing Liu, Tianlang Chen, Wengang Zhou, Yanyong Zhang, Houqiang Li, and Wanli Ouyang. Transvg++: End-to-end visual grounding with language conditioned vision transformer. arviv preprint (xXfc:2266066111 2022. 10, 33
Karan Desai and Justin Johnson. Virtex: Learning visual representations from textual annotations. In CVPR,
2021. 2
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirec- tional Transformers for Language Understanding. In NAACL, 2019. 1, 3, 4, 5, 13
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition al scale. ICLR, 2021. 1, 2, 3, 4, 7, 10
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, Zicheng Liu, and Michael Zeng. An empirical study of training end-to-end vision-and-languuge transformers. CVPR, 2022a. 2, 3, 4, 5, 10, 29
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. An empirical study of training end-to-end vision-and-langungee transformers. In CVPR, 2022b. 8
Zhe Gan, Yen- Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and-langee language representation learning. Advances in Neural Information Processing Systems, 33:
2020. 9, 10, 33
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh, Making the v in vqa matter: Flevating the role of image understanding in visual question answering. In CVPR, 2017. 5, 6, 13, 23
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 13
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022. 1, 2, 3, 4, 5, 6, 10, 13, 22
Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.008: 2020. 2, 3, 7, 8
Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pro-training for vision-language represcntation learning. In CVPR, 2021. 2
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 5
Anya Ji, Noriyuki Kojima, Noah Rush, Alane Suhr, Wai Keen Vong, Robert D Hawkins, and Yoav Artzi Abstract visual reasoning with tangram shapes. arXiv preprint arXiv:2211.66/00 2022. 6
Chao Jia, Yinfci Yang, Ye Xia, Yi-Ting Chen, Zarana Parckh, Hieu Pham, Quoe Lc, Yun-Hsuan Sung, Zhen Li, and Tom Ducrig. Scaling up visual and vision-language representation learning with noisy text supervision. In TCML, 2021. 2
16

---

Published in Transactions on Machine Learning Research (09/2023)
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mcdet-mmodulated detection for end-to-end multi-modal understanding. In CVPR, 2021. 10, 33
Amita Kamath, Christopher Clark; Tanmay Gupta, Eric Kolve, Derek Hoiem, and Aniruddha Kembhavi. Webly supervised concept expansion for general purpose vision models. In ECCV, 2022. 13
Andrej Karpathy and Li Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015. 23
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-languaga transformer without convolution region supervision. In ICML, 2021. 1, 2, 3, 4, 5, 7, 8, 10, 21, 23, 29
Ivan Krasin, Tom Ducrig, Neil Alldrin, Andreas Voit, Sami Abu-El-Haija, Sorge Belongic, David Cai, Zheyun Feng, Vittorio Ferrari, Victor Gomes, Abhinav Gupta, Dhyanesh Narayanan, Chen Sun, Gal Chechik, and Kevin Murphy, Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://githhub. com wwtp:l/ccoooooooeie oppnnmagess 2016. 5,
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 123(1):32-73) 73, 2017. 7,9
Jumnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joly, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. NewrIPS, 2021a. 2, 5, 6, 7, 8, 10, 23
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip:
Bootstrapping language-image pre-training for unified vision-language understanding and generation. arXiv preprinl arXiv:2201.12085 2022. 2, 5,
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint ar Niv::2301 12597, 2023.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and
performant baseline for vision and language. ACL, 2019. 1, 2, 8
Muchen Li and Leonid Sigal. Referring transformer: A one-step approach multi-task visual grounding. NewrIPS, 2021. 10, 33
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning. ACL, 2021b.
Xiujun Li, Xi Yin, Chunyuan Li, Xiaowei Hu, Pengchuan Zhang, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics aligned pre-training for vision- language tasks. ECCV, 2020. 2, 3, 7,
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014. 5, 6, 7, 22, 23, 24
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In CVPR, 2021. 10, 13
llya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In ICLR, 2018. 6, 25
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining tank-agiostic visiolinguistic ropresentations for lsiin-and-langguge tasks. NewrIPS, 2019. 1, 2, 3, 5, 7, 8, 13
Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan L Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In CVPR, 2016. 6,
Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa. In CVPR, 2021. 13
17

---

Published in Transactions on Machine Learning Research (09/2023)
Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXio:2111.097344 2021. 13
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. In ECCV, 2022. 23
Vicente Ordoncz, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. NeurIPS, 2011.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV, 2015. 6, 7, 23
Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. Imagebert: Cross-modal pre-training with large-scale weak-supervised image-text data. arXiv preprint arXiv:2001007966, 2020. 2
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transfcrable visual models from natural language supervision. In ICML, 2021. 2, 13
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In ICML, 2021. 13
Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NearIPS, 2015. 2,
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. IJCV, 115(3):211-255)) 252, 2015. 1, 5, 22
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. arXiv preprint 0tt.:2200000111.. 2022.
6, 13
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, 2019. 7
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACI, 2018. 5
Robin Strudel, Ivan Laptev, and Cordelia Schmid. Weakly-supervised segmentation of referring expressions. arXiv preprint 2rX1::2205.044 04725, 2022.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pro-training of generic visual-linguistic representations. ICLR, 2020. 2,
Sanjay Subramanian, William Merrill, Trevor Darrell, Matt Gardner, Sameer Singh, and Anna Rohrbach. Reclip: A strong zero-shot baseline for referring expression comprehension. In ACL, 2022. 9
Alane Suhr, Stephanic Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. ACL, 2018. 6, 23
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers.
EMNLP, 2019. 2, 3, 13, 23
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigns. orXiv preprint 2xXX::2205.0133.. 2022.
18

---

Published in Transactions on Machine Learning Research (09/2023)
Tristan Thrush, Ryan Jiang, Max Bartolo, Amanpreet Singh, Adina Williams, Douwe Kicla, and Candace Ross. Winoground: Probing vision and language models for visio-linguistic compositionality. In CVPR, 2022. 6, 10
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021. 10
Michael Tschannen, Basil Mustafa, and Neil Houlsby. Clippo: mnage-and-languaae understanding from pixels only. In CVPR, 2023. 2, 22
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, lasks, and modalities through a simple sequence-lo-seqquence learning framework. orXiv preprint ar arXiv:220..030 03052, 2022a. 10, 33
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Unifying architectures, tasks, and modalities through a simple sequenco-to-scquenee learning framework. In JCML, 2022b. 8
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. 2022c. 2, 22
Chen Wei, Haoqi Fan, Saining Xie, Chao- Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training. arXiv preprint Xrxi::2122.00 09133, 2021. 3, 10
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXie preprint orXiv:1609.08144 2016.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In CVPR, 2017. 7
Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim:
A simple framework for masked image modeling. In CVPR, 2022. 3, 10
Hongwei Xue, Yupan Huang, Bei Liu, Houwen Peng, Jianlong Fu, Houqiang Li, and Jiebo Luo. Probing inter-modality: Visual parsing with self-attention for vision-and-language pre-training. NeurIPS, 2021. 2, 5,
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modcling. In ECCV, 2022. 10,
Jiabo Ye, Junfeng Tian, Ming Yan, Xiaoshan Yang, Xuwu Wang, Ji Zhang, Liang Hc, and Xin Lin. Shifting more attention to visual backbone: Query-modulated refinement networks for end-to-end visual grounding. In CVPR, 2022. 10, 33
Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. TACL, 2:67-78, 2014. 5
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosscini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. TMLR, 2022. 2, 8, 22, 23
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L. Berg. Modeling context in referring expressions. In ECCV, 2016. 6, 9
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training Aligning texts with visual concepts. In ICML, 2022.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang. Yejin Choi, and Jianfeng Gao. Vinvl: Making visual representations matter in vision-language models. CVPR, 2021. 2, 3, 5, 7, 8, 21
19

---

Published in Transactions on Machine Learning Research (09/2023)
Chaoyang Zhu, Yiyi Zhou, Yunhang Shen, Gen Luo, Xingjia Pan, Mingbao Lin, Chao Chen, Liujuan Cao, Xiaoshuai Sun, and Rongrong Ji. Sequr: A simple yet universal network for visual grounding. orXiv preprint Xiie:2203.16633 2022. 10, 33
20

---

Published in Transactions on Machine Learning Research (09/2023)
Appendix
Table of Contents
A Pre-training dataset
B Additional experimental results
22
B.1 Natural Language Understanding
22
B.2 Image Classification
22
C Implementation Details for Downstream Understanding & Retrieval Tasks
23
Implementation Details for Downstream Grounding Tasks
D.1 Referring Expression Datasets
24
D.2 Finetuning
24
D.3 Inference....
25
D.4 Additional Math Proof
27
E Different Initialization Methods & Unimodal Pretraining
28
Additional Visualizations
G Additional Results and Error Analysis for Downstream Grounding Tasks
29
H Limitations and Societal Impact
Appendix
This supplementary material has seven sections. Section A describes the details of our pretraining datasets. Section D.3 measures the inference time of different models and highlights the efficioncy of our model. Section C and D describe implementtttin details for ream tasks. Section F shows more visualization examples with more comparisons. Section G includes additional results and error analysis of VLC finetuned grounding tasks. Section H discusses the limitations and socictal impact of this work.
A Pre-training dataset
The statisties of the pretraining dataset is shown in Table A4. Most of the existing approaches, such UNITER (Chen et al., 2020b) and ViLT (Kim et al., 2021), use MSCOCO, VG, GCC and SBU to pre-train their models. We denote this training set as base. To verify the scalability of our model, we follow VinVL (Zhang et al., 2021) to further incorporate VQA, VG-QA, GQA, Flickr30K and OpenImages. As there are some overlaps among VG, MSCOCO and VQA, we exclude all those training images that appear in the downstream tasks via URL matching.
Our extended VLC-for-groonddie pipeline also strikes a superior afffciency--effommanc balance, with visualizations shown in Figure A1 where efficiency indicate by both inference time and parameter count.
21

---

Published in Transactions on Machine Learning Research (09/2023)
Inference time is reported on 1 RTX2080Ti with batch batch_size=1, size=1, averaged across the same 1K randomly selected samples from Refcocog val(umd) (ud) over 5 launches. Since different approaches require different. post-processing for reducing an output Tensor to bbox coordinates, we time execution starting from feeding raw image and text tensors into the pipeline until the four-item coordinates tuple obtained. Bespoke grounding models under comparison oflen involve components overspecialized for the end-goal of coordinate prediction, at the expense of directly optimizing for well-aligned V+L representations. Our results reveal that centralizing the expressive power in single cross-modal encoder benefits V+L unification and efficiency without loss in performance.
B Additional experimental results
In our experiments, we carefully regulate the quantity of pretraining data utilized for two key aspects: the initialization checkpoint, which encompasses sources such ImageNet (Russakovsky et al., 2015), COCO (Lin et al., 2014), WebLI (Chen et al., 2023), and the vision-langnage pretraining phasc. It is essential to underscore that numerous works on vision-language pretraining tend to downplay the significance of dala and computation resources already consumed by their unimodal initializations. Consequently, this inadvertent oversight conceals the fact that their ultimate outcomes necessitate substantially greater resources than would be equitable for a meaningful comparison. In this section, We present supplementary comparisons that demonstrate our model can achieve better or competitive results on ImageNet-1K and GLUF, while utilizing substantially smaller amounts of data. The results implies that our model is friendlier to downstream applications after balancing resource demands and representation quality.
Model
#Params # #Images
PaLI (Chen et al., 2023)
3B-17B
1B
Flamingo (Alayrac et al., 2022)
3.2B-80B
1.8B
CoCa (Yu et al., 2022)
383M-2.18 1B
4.8B
CLIPPO (Tschannen et al., 2023)
316M
10B
SimVLM (Wang al., 2022c)
300M-1B
1.8B
VLC (ours)
307M
5.6M
Table Al: Recent works employ larger training sets and model sizes to emphasize the final performance.
B.1 Natural Language Understanding
We finetune our model on GLUE benchmark. Despite having non of the task-specific supervision and being pretrained on less text, our model still performs competitively on pure linguistic tasks.
Tasks
CoLA QQP
QNLI SST-2 STS-B MRPC RTE
MNLI
SimVLM (Wang al., 2022c)
16.7
90.4
88.6
90.9
84.4
63.9
83.4
CLIP (Tschannen et al., 2023)
6.6
82.7
73.0
86.2
65.0
81.4
53.8
71.8
CLIPPO (Tschannen al., 2023)
55.3
87.9
86.7
94.2
85.8
85.9
59.2
82.3
VLC (ours)
41.8
89.7
87.0
89.7
87.0
88.6
59.6
78.8
Table A2: Our VLC performs competitively on the GLUE benchmark dospite that it is pretrained on less text without task-specific supervision.
B.2 Image Classification
We present a comparative analysis of performance in the ImageNet- 1K image classification task. Despite utilizing a substantially reduced image dataset, our VLC surpasses the achievements of existing works, thus demonstrating the efficacy of our model. We adhere the hyperparameters specified in MAE (He al., 2022) without any additional tuning.
22

---

Published in Transactions on Machine Learning Research (09/2023)
Model
# Params #images ImageNet- 1K (finetune)
CLIP (Mu et al., 2022)
350M
15M
81.0
SLIP (Mu et al., 2022)
350M
15M
84.2
CoCa (Yu et al., 2022)
393M
3B
84.9
VLC (ours)
307M
5.6M
86.3
Table A3: Our VLC demonstrates superior performance in the ImageNet- image classification task, despite employing significantly smaller image data.
Dataset
MSCOCO
VG
GCC
SBU
VGA
GQA
VG-QA
Flickr30K
Openlmages
# Images
113K
100K
2.95M
860K
83K
79K
87K 931K
29K 145K
1.67M
# Text 567K
769K
2.95M
860K
545K
1026K
1.67M
Table A4: Statistics of the pre-training dataset
C Implementation Details for Downstream Understanding & Retrieval Tasks
For downstream understanding and retrieval tasks, we fino-tune our model with a learning rate of 54 for 10 epochs. We use 480 x 480 the input image resolution for the VQA task and 384 x 384 for NLVR2 and image-text retrieval tasks.
Visual Question Answering (VQA (Goyal et al., 2017)). Given input image and question, the VQA task is to predict an answer from the visual content. We conduct experiments on VQAv2 dataset (Goyal et al., 2017) that is built on MSCOCO. It contains 83K images for training, 41K for validation, and 81K for testing. Following previous work (Tan & Bansal, 2019; Chen et al., 2020b; Li et al., 2021a), we use the training, validation splits and additional question-answer pairs from Visual Genome while reserving 1, 000 validation imagc-question pairs for internal validation. We report performance on the test-dev and test-std splits. We follow the standard practice (Kim el al., 2021) to convert the task to multilabel classification task with 3, 192 answer classes.
Natural Language for Visual Reasoning (NLVR? (Suhr et al., 2018)). Given triplet of two images and a description, this task is to predict whether this description describes pair of images. Following previous work (Kim et al., 2021; Chen et al., 2020b), we use the pair method which treats one input sample as two image-lext, pairs by repeating the text twice. Each pair is passed through our model and we take the concatenation two pooled representation [CLS] from our model as the representation of one input sample. Similar to the scttings of the VQA task, we use a 2-layer MLP with a hidden size of 1, 536 to adapt VLC the 2LVR² task.
Image-Ters Text Retrieval. Image-Text retrieval contains two subtasks: image-to-text retrieval (TR) and text-to-image retrieval (IR). We evaluate our pre-trained models the Karpathy splits (Karpathy & Fei-Fei, 2015) of MSCOCO (Lin et al., 2014) and Flickr30K (Plummer et al., 2015) in fine-tuning settings. MSCOCO contains 123K images, and each image has five corresponding human-written written captions. We split the data into 82K/5K/50 /5K training/ caaiiing/validatiii validaatioonnn ion /est images. To be consistent with previous work (Kim et al., 2021; Chen et al., 2020b), we use the additional 30K images from MSCOCO validation set to improve the performance. Flickr30K contains 31K images with five captions for each image. We split the data into 30K/IK/IK as the set.
training/ training/validation,,
Implementation Details for Downstream Grounding Tasks
The main text has briefly argued that appropriate supervision enables generalizable and versatile representa- tions which naturally translate downstream grounding performance without tailored design. This scction describes our approach of adapting the Transformer backbone to performing grounding tasks in detail. We conduct grounding experiments Refcoco/ //gg. Namely, Section D.1 provides dataset details, Section D.2
23

---

Published in Transactions on Machine Learning Research (09/2023)
OFA Jarge
450
UniTAB
400
Inference Time (ms)
Parameters (M)
VL--T5 .QRNet
OFA _lrrge
230
200
Mdetr-R101 e UniTAB
200
VL-TS
OFA_ base
@RNNet
TransVG
OFAA base
SeqTR
Mdetr-ENB3
100 TransVG
Mdetr-R101
130
Mdetr-ENB3
VLC base
SeqTR
VLC base
Performance (Refcocog_vall
Performance (Refcocog_vall
Figure A1: Inference time and parameter counts of different Referring Expression models. Our VLC Base achieves competitive performance with considerably less inference time (left) and parameter count (right)..
introduces the process of converting bounding box annotations to patch regions for supervised finetuning, and Section D.3 describes how the resulting affinities can be re-interpreted as either a mask or a bounding box.
D.1 Referring Expression Datasets
The referring expression task feeds model with an (image, region description) pair, then expects localization output as either bounding box (bbox) or pixel-level mask corresponding to the description. It tests for visual grounding and reasoning as the model indicates which part of the image is referenced by the text. Note, a mask output is more challenging to produce since a mask can adopt various shapes and requires a dense prediction in the original image's resolution.
We three Referring Expression datasets: Refcoco, Refcoco+, and Refcocog. All of which MSCOCO (Lin al., 2014) training images. Refcoco and Refcoco+ collected from human annotators participating in "Referlt" Game, where one player selected regions in an image referenced by text description provided by the other player. Where they differ is that Refcoco+ disallowed players from using ocation-basch expressions (e.g. "person to the right"), thereby being purely appearance-based. Refcocog was collected in a non-game setting but contains more linguistically complex descriptions.
Bounding boxes are evaluated by Interseation-over-Unioo (IoU), while masks are evaluated by pixel-level ToU. To compare with other relevant models, we follow the literature by reporting ToU acc, which is the proportion of testing samples which the model achieves an loU > 0.5.
D.2 Finetuning
Our approach builds on the notion of an affinity map. Formally, we define the affinity map, A, as the cosine distance between all Canguage tokens and Visual patches.
Ar,p cos(LL, Vp))
24

---

Note, that the domain expected by BCE is the full real line, but our affinities are computed via cosine similarities and are therefore bounded to [1,,]]. Fmpirically,, we found that re-scaling the input by k = 2 improved performance due to a better utilization of the [0.1] output range. However, performance decreases Published in Transactions on Machine Learning Research (09/2023)
for T tokens and P patches. This resulting matrix 1| provides normalized score for the relationship between every token and patch.
Supervision Signals For supervision, we translate bounding box annotations to sets of patches and phrases simply correspond to their indices, with [CLS] being the initial index O. The ground truth affinity scores can be obtained from labeled datasets where annotators selected bboxes corresponding to phrases in an (image, sentence) pair. Thus, for an (image, sentence) pair, we have set of annotated (bbox, phrase) pairs, from which we would like to get T x P matrix of affinity scores. Note that the full image has dimensions H W and will be divided into vPx x vP grid.
For a given annotated (bbox, phrase) pair, if the lexical token L, belongs to the phrase and patch Vp overlaps with the bbox, we set At.p in the ground-truth affinity map to be amount the patch overlaps with the bbox.
Patch n BBox
A
(8)
Patchh]
If a bbox corresponds to the entire sentence, we set ground-trutt truth affinity scores for the 0-th token, [CLS].
Hyperparameters We opt for a higher resolution (384) and smaller patches (16) since the patch size will determine the gramularity of our predictions. This yields a final 24 grid. We finetune VLC on the combined Refcoco/+ +/g training sets for 50 epochs with AdamW (Loshchilov & Hutter, 2018) optimizer and 54 learning rale.
Finetuning Objective We optimize our model with a binary-cross-entropy (BCE) loss between the ground-truth and predicted affinity scores.
Loss
OCE(At.p,...)
(9)
TTPE
We intentionally avoided the use of a softmax because normalization would force competition across patches and affect their individual judgments. Each token-patch pair relationship is independent during loss calculation.
with a larger k because it tends to over-sharpen the sigmoid's decision boundary, leaving less "wiggle room' for the raw logits.
D.3 Inference
The forward pass through the Transformer outputs single tensor of shape L L 768, or more accurately, the concatenation of two tensors of shape 768 and Px 768 for the encoded Canguage and Visual sequences, respectively. During inference, these matrices are used in Equation 7 to compute the predicted affinity matrix Á. Next, we describe how a mask or bounding box can be algorithmically derived from a predicted affinity matrix Å. We emphasize that this procedure requires no additional learned parameters and algorithmically efficient.
Since rows in Aacto associated with a phrase can be pooled into single row and reshaped into /P x vP, we henceforth refer Arcto to such a []1,1)'PxxV matrix with spatial correspondence the image, albeit with coarser resolution. Then, binary mask can be simply derived by bilinearly interpolating A' back to the original resolution, and binarizing the values with threshold.
The procedure for deriving bbox from Åiat is more complicated, which we detail below. We formulate bbox prediction as a search problem. A bbox is denoted by = (z, y, w, h), satisfying 0 < x < I + w < W, S y < y + < H. The goal is to scarch for B given Ai such that B best represents a rectangular region Ar trics to highlight (Figure A2, Left). To solve the search problem we need to decide the following: the search space, the search order, and the search criterion.
25

---

Published in Transactions on Machine Learning Research (09/2023)
Search Space: Al bboxes lte on grillines
Search Criterion:
loU-style
Imogge Haight
0.21
0.5
Search Order: From large smail
Image Widh
Patch size
Edges to DUS!
Figure A2: Inference procedure overview for "Affinity Map + BBox", which is formulated as a search problem. Left: we define the search space to contain all candidate bboxes that lie on gridlines. Right: our algorithm greedily iterates over bboxes following the search order of large + small, and terminates once the search criterion longer improves.
Search space Thcoretically, we have an infinite number of boxes search over which can be as large as the input image or as small as nearly single point. We constrain our search to only those boxes whose edges lie on gridlines, with step t denoting the unit between two gridlines.
Search order We start with box encompassing the full image and greedily search for progressively smaller boxes.
Search criterion We propose satislying criterion M that guarantees equivalence between Abste and the theoretically optimal bbox (B). In the case where all values in Andt are binary, this reduces to calculating a standard IoU (Intersection-oveey Unnn))-nnn))))e----nnn)))))))))) In practice, values will be real-valued. We overload the notation to handle the generalization from binary to real values in Aucte- Intersection is defined as the sum of the active values within Union is equal to the sum of three terms, namely the active values inside B, the values that should have been active inside B and the active values outside B. The first two terms in Union add up [B]. As single equation this results in
A n n M
(10)
|B| + Ar n-B]
In practice, this is an easy calculation, as the intersections and negations simply correspond to summing the values inside and outside the candidate box. Further, summing (in leiu of counting) trivially generalizes to the continuous valued activations.
We provide additional proof of the equivalence to optimizing F1 in the supplementary. We will note, there is potential hyperparameter which will call C that defines the desired "tightness" of the box.
Next we leverage the the scarch space, order and criterion (M) in Algorithm to predict bounding boxes. Initializing B to surround the entire image, our PUSH algorithm greedily iterates over progressively smaller B and terminates when M(Ar B3) no longer improves. Within each iteration, there are "push' attempts on the edges of B, one at time, by t-amount inwards according to an order O E Permutationn(T. L, B, R)). For example, O [T, R. B, L] would translate to clockwise progression of tightening edges from the Top Right Bottom Le A push operation is successful if and only if it increases M. Otherwise, the edge is left unchanged. The algorithm terminates when it encounters unsuccessful push attempts all four edges in a row. Figure A2 visualizes PUSH execution.
For example, C 0.5 |Atre would produce bboxes systematically smaller than C [A The supplementary discusscs about where derived from. In this work, we leave the scaling and allow future work analyze how C would influence the returned value of A Algorithm.1
26

---

Published in Transactions on Machine Learning Research (09/2023)
Algorithm 1: PUSH
input :heatmap Abatu image size (W, H), step mcasure M (A BB + R,
order O E Permulation((: L, B, Rl) output: : B: box coordinates (z, y. w, h) s.t. 0 x x w S W and 0 s s
(z, y: w. w,,)., h) 00,0, W,W, H)
B - (z, y, w. h)
while w > 0 and h > 0 and move & Folse do
for e E 0 do
if e == then B' t (I, + t, w, , h - t)
else if e - B then B' + (1, U/ W; 1)
else if
R then B' (I, y,
h)
else B' (x + 1, t, h)
if M( actor B²)= > M(A (3) then
B + B
(z, y, W, h) + B
move t True
end
end
end return B
In our experiments, we use values P 24, W - H 384 and 1 patch size 4. A known limitation of our approach is highly discontiguous concave affinity patterns. We did not observe this condition in our setting. We provide empirical evidence that our "Affinity Map = BBox" formalization is effective but leave a mathematically rigorous investigation to future work.
D.4 Additional Math Proof
In the section above, we derive the search criterion as the Intersection-over-nniio (IoU) between Aiato and B:
A n B
M
(11)
[B]
Ar n-
We will further argue that optimizing for IoU equivalent to optimizing for F1 the context of our PUSH algorithm. Using the same notation, we write down F1 between Aietv and B as follows:
2| n
M
B + A aotu
0B) + A n n-B)
(12)
A n
B + A
In order to prove the equivalence of the two criteria, M and F we simply need to show that when a push operation causes one critcrion to increase, the other critcrion must also increase well.
Assume single push operation reduces ( nB] from an to 02 and reduces B from b to b2. Let C = ( which holds constant throughout the search. Eq. 13 shows that M increases iff M increases.
Empirically, we find that the order of [L, B, R, does not affect performance. We the Refcocog_vvlluuee 10 times, randomly permuting the order of [L. B, R, T] at each PUSH iteration. For 90% samples the algorithm returned the same predicted bbox all 10 times. For the remaining samples, the 10 returned bboxes have joint Io 0.000.95.0.07.
bbox inference procedure
We ignore the scaling by the numerator of F1 because constant scalar will affect search procedure.
27

---

Published in Transactions on Machine Learning Research (09/2023)
Q1
a2
<
1i + C b2 C
+ aib2 + 01 C azb1 a²C
(13)
aiby + +1 a10b azbi + a2 0142
a
02
4 + d1
by + a2
E Different Initialization Methods & Unimodal Pretraining
The figure A3 compares VLC with different initializations. We show that most of performance gain comes from VL pretraining (VLP), while MAE initialization improves the training speed. (1) Supervised INTIK. It has a strong prior (67.19% before VLP), but flat learning curve (+3.36% gain). We hypothesize that such initialization is more biased to predefined INTIK classes, and inconsistent with the visual embedding space learned by MIM. This makes the model difficult to learn multimodal representations by simply switching one objective to another. (2) Random. It has the most substantial improvement (+12.23% gain), but still lags behind MAE initialization. We hypothesize that 200K steps (80 epochs) are not sufficient for images. As comparison, MAE trained with 1600 epochs on INT1K. More training may mitigate the gap. (3) Neither models, random nor MAE initialization saturated.
72.98
72.5
71.69
70.55
70.0
69.52
69.62
70.05
VOAv2 accuracy (%)
68.53
67.5
67-28
67.32
$5.45
65.0
63.81
62.5
60.0
Random initialization
MAE initialization
57.5
$7.82
Supervised INTIK initialization
25
50
75
100
125
150
175
200
Number of training steps (K)
Figure A3: Comparisons of our VLCBase with different initializations.
We show that MAE initialization
improves the training speed.
Unimodal pretraining initializes the representation space, exposing collapsing unique propetties/cussen clusters. We will expand this analysis, but also include three additional benefits. The table A5 compares unimodal pretraining approaches on VQA. (1) There is general trend that better pretraining provides stronger downstream priors (METER with Swin/INT21K: 72.38% before VLP), (2) Pretraining can improve the multimodal training speed. As the image/texx modalities have different learning speeds. MAE and BERT require 1600 vs epochs on INT1K and text corpus, respectively. Better image training creates more meaningful representations which allows for "fast mapping' in language acquisition (Fig. 5), and (3) Primary performance gains (+ (0.2%) come from VLP which demonstrates the effectiveness of our model.
28

---

Published in Transactions on Machine Learning Research (09/2023)
Models
VLClarge (Ours)
ViLT (Kim et al,, 2021)
METER (Dou et al., 2022a)
Initialization
MAE
VIT(21K)
Swin(2IK)/Robas
w w/o VLP
66.75
67.19
72.38
w/ VLP
76.95 (+10.2)
71.26 (+1.007
76.43 (+4.05)
Table A5: Comparisons of the benefit of different unimodal pretraining on VQA. We show that most of our performance gain comes from VL pretraining.
F
Additional Visualizations
We provide additional visualizations for patch-token alignment resulted from pre-training. Specifically, we choose tokens be nouns in Figure A4, adjectives in Figure A5, and verbs in Figure A6.
The inductive bias of organizing patch-token affinity patterns into bounding boxes can be correctly learned through finetuning, seamlessly adapting the pretrained model to localization tasks. However, in downstream applications when only single object is expected to be aligned to a phrase (i.c. span of tokens), model predictions are occasionally distracted by attributes or relations. Table A7 uses the same input image to illustrate the model's sensitivity to varying attributes and relations (after finetuning). The original referring expression was "man walking with purple shirt woman', to which the model gave an incorrect answer. We identify two potential sources of error. First, the model might not be able to detect the woman or the man because they are in the background. Second, the model was able to detect background entities, but the reasoning step might fail to distinguish between entities. We linguistically vary the input expression in order to single out the error source.
Table A7, Middle, indicates that detection error was not the cause of an incorrect prediction, because both the woman and the man can be successfully detected via more accurate and distinct attributes. Table A7, Right suggests that the reasoning step could go wrong for subtle reasons. For example, viewing the scene as a 2D image, the blue shirt man indeed walking next the purple shirt woman. But it requires muanced reasoning about the depth the head direction to infer that the blue shirt man could not be with the purple shirt woman. Moreover, both "black shirt" and "purple shirt' occurred twice in the image. Therefore, they could possibly confuse the model. On the other hand, "white cap" and "yellow shorts" are distinct attributes, which easily help the model with making the correct choice.
G Additional Results and Error Analysis for Downstream Grounding Tasks
Table A6 compares the Referring Expression Comprehension performance on both the validation and test splits. Results on the test splits exhibit a similar trend as validation splits. Our performance is superior to the previous specialized models, on par with moderate-size unified models, but is lower than those very largest and slowest unified models.
Next, we analyze commou prediction errors which could inform future research on the unsolved difficulties of Referring Expression tasks. Table A8 visualizes the error-prone cases on Referring Expression Comprehension. The model blunders when the target entity is specified through complex attributes or relations. The model also occasionally confuses a head noun with its modifiers, or gets distracted by an irrelevant entity sharing a mentioned attribute. Mistakes can also stem from inadequate text-reading ability from visual stimuli.
Table A9 shows a performance breakdown according to four potential sources of difficulty. Table A9a suggests that the model particularly struggles when referring expression contains multiple noun chunks. We believe that the number of noun chunks is reasonable indicator of difficulty because it also correlates well with the sentence length and linguistic complexity. Table A9b suggests that the model is less performant when it encounters an unfamiliar head noun. We parse the expressions and extract noun chunks via SpaCy. A token identified as the head noun its part-of-speech tag is one of ['NOUN; 'PROPN' and its role in the dependency tree is one of [nssubj,, 'ooo"". The head noun frequencies are counted across Refcoco/+/g val.
These heuristics occasionally fail to identify head noun or identify more than one head noun < 5% of the cases.
29

---

Published in Transactions on Machine Learning Research (09/2023)
Caption with focus
Original Image
ViLT
VLC
hawk is perched on a metal bar
GANINAA VACIFER
Bench Gr nde
gift wrapped with a ribbon sits on a table with knife
A plate with pancakes, syrup, grits, and butter
There is colorful parachute in the sky
Figure A4: We visualize patch similarities with nouns OOD images. Note that ViLT is often picking up relevant features but has the strongest correlation with single, presumably predictive patch.
Table A9c plots the performance against the area of the ground-truth bounding box. We observe that the performance drops drastically on extremely small regions. Table A9d plots the (zero-out) segmentation performance against the convexity of the ground- between the area of the polygon and the area of its convex hull. Note, the presence of holes inside a mask or non-contiguity could also lead a small convexity score. Thus, our findings reflect ample room for improvement. localizing small and less convex regions.
polygon mask. We measure convexity as the ratio
30

---

Published in Transactions on Machine Learning Research (09/2023)
Caption with focus
Original Image
ViLT
VLC
A red fire hydrant in front of skyscraper
A monarch butterfly lands pink flower.
A small orange and blue ladybug sitting on long green leaves
A brown and white dog
is holding a yellow Fris- bee
Figure A5: We visualize patch similarities with adjectives on OOD images. VLC produces more accurate and comprehensive alignment. Note that the lady bug is correctly associated with the relative size small. Future work would ideally investigate how models align visual entities to more abstract and comparative concepts in the language domain.
Limitations and Societal Impact
In our work, we train our model with image-text pairs 4M/5.6M unqiue images. Limited to our computation resource, it is unclear how the model performance will be with further scaling, In addition, we only focus on images with English descriptions. The generalization ability to other languages left for the futher work.
While paper shows promising results vision-language pretraining and the scaling of training data and model size, additional analysis should be undertaken to assess the quality of pretraining web data before further deployment. Because web data may contain gender/race /racial bias, hate speech, private information and other unsuitable images. Only optimizing the performance may have unwanted model behaviors.
31

---

Published in Transactions on Machine Learning Research (09/202))
Caption with focus
Original Image
ViLT
VLC
A person who is hitting ball with bat.
A person holding a cell phone their hand
A green boat floating top of a body of water
joyże
an orange and white cat sitting on a bed staring at the viewer
Figure A6: We visualize patch similarities with verbs on OOD images. Note that aligning verbs to static images is a slightly strange task, but our model still manages associate key perceptual indicators a verb's semantics. For example, holding aligned to person's 's hand and staring picks up the cat's eyes.
32

---

Published in Transactions on Machine Learning Research (09/202))
Model
Time
Params
Refcoco
Refcoco test.A
Refcocog (umd)
vsl
testA
test
val
test.
val
test
VL-T5 (Cho et al., 2021)
5.9x
290M
71.2
71.3
Trans VG (Dong ct al., 2021)
2.8x
169M
80.83
83.38
76.94
68.00
72.46
59.22
68.71
67 .9.8
Modular
TransVG (Deng al., 2022)
86.28
88.37
80.97
75.39
80.45
66.28
76.18
76.30
QRNet (Ye al., 2022)
5.5x
273M
84.01
85.85
82.34
72.94
76.17
63.81
73.03
72.52
RefTrans (Li Sigal, 2021)
85.65
88.73
81.16
77.55
82.26
68.99
79.25
80.01
Soq (Zhu et al., 2022)
4.4x
108M
83.72
86,51
81.24
71.45
76.26
64.88
74.86
74.21
UNITERLaage (Chen al., 2020b)
371M
81.41
87.04
74.17
75.90
81.45
66.70
74.86
75.77
VII.LALaage (Gan al., 2020)
371M
82.39
87.48
74.84
76.17
81.54
66.84
76.18
76.71
VI.Cruo (ours)
1.0x
110M
86.67
89.15
81.99
77.44
82.57
69.71
80.43
80.22
OFAxtdium (Wang et al., 2022a)
98M
85.34
87.68
77.92
76.09
83.04
66.25
78.76
78.58
MDETR-R101 (Kamath et al., 2021)
3.0x
185M
86.75
89.58
81.41
79.52
84.09
70.62
81.64
80.89
Uniffed
MDETR-ENB3 (Kamath et al., 2021)
2.6x
153M
87.51
90.40
82.67
81.13
85.32
72.96
83.35
83.31
UniTAB (Yang et al., 2022)
19.4x
198M
88.50
91.06
83.75
80.97
85.36
71.55
84.58
84.70
OFA (Wang al., 2022a)
6.0x
180M
88.48
90.67
83.30
81.39
87.15
74.29
82.29
82.31
OFAtarg (Wang et al., 2022a)
11.5x
470M
90.05
92.93
85.26
85,80
89.87
79.22
85.89
86.55
OFA (Wang et al., 2022a)
930M
92.04
94.03
88.44
87.86
91.70
80.71
88.07
88.78
Table A6: Results on Referring Expression Comprehension. Models compared along two axes: efficiency- performance and efficiency-versatiiity Our model beats competes with other models, while being much lighter and faster. The Params column counts all parameters required for inference, including the upstream Rol extractor (if nceded), the main architecture, and specialized output heads.
33

---

Published in Transactions on Machine Learning Research (09/2023)
Input image
Predicted (incorrect)
Predicted (correct)
man walking with purple shirt
man in white cap
woman
man in black shirt walking with purple shirt woman
man yellow shorts walking with purple shirt woman
woman in purple shirt
woman in purple shirt and black shorts
woman walking with a man
women walking with man in yellow shorts
Table A7: Qualitative demonstration: The model is able detect small entities in the background. But the model's success at distinguishing between entities relies accurate and distinct attributes or relations. Here juxtapose changes to phrasing addition of a clause.
34

---

Published in Transactions on Machine Learning Research (09/202))
Input image
Predicted affinity map
Predicted bbox reduced
Predicted vs. truth
Ground-
man walking with purple shirt woman
The model failed to disambiguate via relationship basod attributes.
cake layer by lork end
The model tried to highlight the fork end while the head phrase is "cake layer'.
black and gold tie anbrisc bierce book
The model was distracted by gold scissors on black tie.
The model struggled text-reading from image, thus predicting at extremely low confidence.
Table A8: Qualitative demonstration. Common error types include 1) failure to resolve relationship-basel attributes, 2) incorrect head-noun identification, 3) distraction by an irrelevant object sharing mentioned attribute, and 4) inadequate text-reading from visual stimuli.
35

---

Published in Transactions on Machine Learning Research (09/2023)
6.1
0.1
Nvg bXL
50300
282-50
Number of Noun chunks
Head Noun
d
0.2
6.4
Nog Iou
Rung Sengmentatii
000 826
625
EJ 04 249 Moe
10 835 249 OF 8/6 646 85s
BaxArca las propersion of the full Image)
Convexity
Table A9: Performance breakdown according to different indicators of the difficulty level. We observe that the performance drops when a) the referring expression contains multiple noun chunks, b) the referring expression has rare head noun, c) the ground-truth bounding box has a small area, d) the ground-truth polygon mask is less convex. Statistics computed across Refcoco/+/8 val. Shades indicate the number of examples in each bin. The horizontal line marks the global performance averaged over all bins.
val.
36


Reviews:
[{"review_id": "t3Rw36heWY", "reviewer": "Reviewer_PotT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n[S1]: Clearly written: The paper is clearly written and contains sufficient details and thorough descriptions of the experimental design. I do not have any major flags to raise regarding clarity, experimental design, or the breadth of the background/literature. \n\n[S2]: Technically sound and of interest to the community: The paper clearly satisfies both big-ticket criteria for TMLR, i.e., Technical soundness and interest to the community. To this reviewer, the simple-yet-effective model design, thorough experiments, and great writing all clearly show the technical soundness of the paper. Similarly, while there are caveats (see below), the core set of findings in indeed novel and should be of interest to (at least some) member of the community.\n\n\nWeaknesses:\n\n[W1]: Core premise unclear: The core premise of the paper appears to be that it is possible to train transformers from text alone unlike prior methods which requires training with region/box annotations and/or imagenet trained initialization. As a researcher also working in this field, I have not interpreted the state of multimodal pretraining in this light. To me, the choice to use box annotation and/or Imagenet pretraining was to speed up training and/or minimize training stability (i.e., done for practical reasons) and not because the community collectively thought that training from scratch was not possible. However, I do admit, after this manuscript that there does seem to be a dearth of \"training from scratch\" models. So, the findings are still valuable. Furthermore, the paper does show that it is possible to get competitive results without the use of additional data, which is indeed valuable. However, I would like to see more discussion/proof about the core premise and exactly what the authors are claiming so that the reviewers can evaluate it in context.\n\n[W2]: Rather niche/cherrypicked set of conditions: To this reviewer, the final set of conditions seems to be ever-tightening to the point that it becomes quite rigid/artificial. \n- It is possible to train transformers with text-only supervision \n(So does Virtex, https://arxiv.org/pdf/2006.06666.pdf), which has a transformer for the text side but not of vision side which still uses CNNs. \n\n- So, revising that, I guess the paper is claiming that *both* vision and text transformers can be trained with text only. But, CoCa and CLIP also show that too. Then, the paper makes another clarification that they use >10M data. To this point, CLIP (and variants) and CoCa both do continue to scale with more data, but both have been shown to *start* working with a much smaller amount of data (~15m in DeCLIP, SLIP, etc..12M in MERU ). \n- So, I think the paper is showing that vision and text transformers can be trained without Imagenet initialization on a small data regime. \n\nIs this understanding correct?\n\nTo me, this is neither made adequately clear in the paper nor do I see a broad appeal for this set of conditions. Why are they important? Together with W1, I would say that this lack of motivation/clarity about the scope (also W1) of the paper is its biggest weakness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a (pre)training method for vision language transformer that does not require a pretrained image-encoder trained either on Imagenet or \"dense\" labels such as bounding box annotation, region annotation etc. The new method is based on masked __ modeling for both images and text which is shown to be effective in learning good representation without any pre-pretraining required. Other interesting properties also emerge, such as the fact that the \"traditional\" model seems to provide a good headstart but also a lower \"ceiling\" compared to the proposed model when the network sizes grow. The results show competitive or better results compared to \"traditionally\" trained multimodal transformers when compared against a wide range of existing methods.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "[Q1]: Ref W2, Please clarify the scope and importance of being able to do a specific set of conditions (train without Imagenet pretraining on a small amount of data). Please also clarify what conceptual differences are there compared to CLIP-like (Hyper or CoCa-like models. \n\n[Q2]: Why is >10M data treated as a separate thing? Is the claim that the proposed method is more scalable? Is the paper claiming faster convergence on smaller data? Is it better than initializing from a previously trained model such as CLIP? Please clarify what \"value proposition\" is offered by the paper.\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "u2AP1Ui6f5", "reviewer": "Reviewer_SRjs", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Extensive experiments are conducted to verify the effectiveness of the proposed method. The proposed method can lead to good grounding performance and benefit compositional reasoning.\n\n- The proposed method is simple yet effective. The model is also more efficient compared to existing ones.\n\nWeaknesses:\n\n- The claim in the Introduction \"Our Vision-Language from Captions (VLC) model matches or outperforms nearly all vision-language transformers\" is not well-supported. There are quite a few recent works on learning vision-language Transformers including [r1, r2, r3]. Since the existing methods considered in the paper are a bit out-of-date, I think it would be better to modify some claims accordingly. \n\n[r1] UL2: Unifying Language Learning Paradigms\n\n[r2] PALI: A JOINTLY-SCALED MULTILINGUAL LANGUAGE-IMAGE MODEL\n\n[r3] CLIPPO: Image-and-Language Understanding from Pixels Only\n\n- Some conclusions are a bit confusing. The paper claims the proposed method benefits from avoiding a vision-only processing stack and avoiding priors from ImageNet pretraining. But the results in the paper show the proposed model can achieve comparable performance without these designs or pretraining. Is it possible that the model can be stronger if a stronger (and large) vision tower is added to the proposed framework? Considering there are many exciting progresses like CLIP on learning better backbone models, standing on the shoulders of these powerful models instead of intentionally avoiding using them may be a promising solution to solve vision problems. \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a method to learn vision-language Transformers with image-text pairs and a mask modeling pretext task. Without using classification or detection labels, the model achieves good performance on various vision tasks. The paper also shows a clear performance improvement over ViLT. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please refer to my comments above. I think it would be beneficial to include and discuss some more recent methods. ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "bnUWpGJCYH", "reviewer": "Reviewer_2YtN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "\nI feel like the abstract is incorrect. \"Existing work, whether explicitly utilizing bounding boxes\n(Chen et al., 2020b; Tan & Bansal, 2019; Lu et al., 2019) or patches (Kim et al., 2021),\nassumes that the visual backbone must first be trained on ImageNet (Russakovsky et al.,\n2015) class prediction before being integrated into a multimodal linguistic pipeline\"\n\nThere are many works that use weak labels or captions for training a ViT backbone. For example PaLI (Chen et al ICLR 2022) uses caption data to train the vision model. CoCA (Yu et al 2022) and SimVLM (Wang et al 2021) also use captioning data to train the ViT. These works are not addressed in the paper. Works like CLIP (Radford et al 2021) and Flamingo (Alayrac et al 2022) also do not use imagenet pre-training.\n\nThe claimed contributions:\n\" 1. Smaller and faster at inference, 2. Avoiding use of ROIs, and 3. Not leveraging\nobject-level supervised labels for pretraining.\"\n\nAren't that strong. Using ROIs, is no longer standard, pretty much all recent works have moved away from using ROIs and object-level supervision. \n\nThe approach itself (e.g., Figure 2), is nothing novel. It consists of modality encoders, concatenation, then a transformer followed by multiple output task decoders. There are many works using this type of model. This work, with its pretraining tasks, seems very similar to many previous works (e.g., arXiv:2205.00949).\n\nMethods like CoCa are missing from Table 2. With so many missing related works and missing comparisons, it is hard to tell if the state of the art comparisons are accurate. Table 3 results are quite weak as well, especially since the main advantage of using only weak labels is the ability to scale to larger datasets, the comparisons with other similar methods (e.g., Pali, coca, flamingo, etc) should be better.\n\nThe localization results are more interesting, showing the models ability to adapt to a localization task. I think focusing on this aspect of the paper would be better, as it is more unique and different from many previous works.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a model for vision + language learning using only captioning data. The approach is evaluated on a set of standard tasks and datasets. ", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think the claims should be revised and clarified with respect to the prior works. The current claims aren't entirely true as many recent works use similar models, tasks, and data for vision + language learning without using ROIs or object level data.\n\nMany related works are omitted and should be discussed (coca, flamingo, simvlm, etc).", "broader_impact_concerns": "None", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The submission presents a method building on masked autoencoders called Vision-Language from Captions (VLC) for training vision-language models using only caption data.

Given the modifications promised by the authors, the paper would meet the bar for acceptance in terms of empirical support of the claims made and appeal to TMLR's community. Acceptance is therefore conditional on the following modifications appearing in the camera-ready version:

* (2YtN) Edit the abstract to make our contributions clear
* (2YtN) Clarify our motivation of controlling for the dataset size and initialization
* (PotT, SRjs) Add comparisons with CoCA and SimVLM on model and training data scales
* (SRjs) Add conceptual comparisons with PaLI, CLIPPO and Flamingo in related works
* (PotT) Add conceptual comparisons with contrastive models such CLIP, CoCA in related works
* (PotT,2YtN,SRjs) Add comparisons with recent models on GLUE evaluations
* Add missing references such as VirTex, UL2, CLIPPO
* Address Reviewer 2YtN's concern that "the linear probing [in the added results on ImageNet] show [CoCa/CLIP/etc.] to be better" and determine whether this is a case of overfitting or better hyperparameter tuning in the finetuning case.
Decision: Accept (poster)