GLIPv2: Unifying Localization and Vision-Language Understanding 
Authors: Haotian Zhang
Pengchuan Zhang
Xiaowei Hu
Yen-Chun Chen
Liunian Harold Li
Xiyang Dai
Lijuan Wang
Lu Yuan
Jenq-Neng Hwang
Jianfeng Gao
Venue: NeurIPS 2022 Accept (2022)
Keywords: region-aware
vision-language
open-vocabulary object detection and segmentation
phrase grounding
VQA
image captioning

Abstract: We present GLIPv2, a grounded VL understanding model, that serves both localization tasks (e.g., object detection, instance segmentation) and Vision-Language (VL) understanding tasks (e.g., VQA, image captioning). GLIPv2 elegantly unifies localization pre-training and Vision-Language Pre-training (VLP) with three pre-training tasks: phrase grounding as a VL reformulation of the detection task, region-word contrastive learning as a novel region-word level contrastive learning task, and the masked language modeling. This unification not only simplifies the previous multi-stage VLP procedure but also achieves mutual benefits between localization and understanding tasks. Experimental results show that a single GLIPv2 model (all model weights are shared) achieves near SoTA performance on various localization and understanding tasks. The model also shows (1) strong zero-shot and few-shot adaption performance on open-vocabulary object detection tasks and (2) superior grounding capability on VL understanding tasks. 

GLIPv2: Unifying Localization and VL Understanding
Haotian Zhang" Pengchuan Zhang 21
Xiaowei Hu², Yen-Chun Chen', Liunian Harold Li Xiyang Dai Lijuann Wang", Lu Yuan², Nenq-Neng Hwang', .Jianfeng Gao
Universstty Washington, MMta AI, Microsoff, *UCLA
(baotin, hwang Ruw. edu, peagchuanzhengeeb.. con, liunian harold. liice.ucll edl.. (Xiaoeei Bu, Yen Yeu-Chun Chen, Xiyang: Dai, 11 juanw, Juyuan, jagaoo Swicrosoft.ccmm
Abstract
We present GLIPv2, a grounded VL understanding model, that serves both local- ization tasks (e.g., object detection, instance segmentation) and Vision-Language (VI.) understanding tasks (e-g.. VQA, image captioning). GLIPv2 elegantly uni- fies localization pre-training and Vision-Language Pre-training (VLP) with three pre-training tasks: phrase grounding as VL reformulation the detection task, region-word contrastive learning as novel region-word level contrastive learning task, and the masked language modeling. This unification not only simplifies the previous multi-stage VLP procedure but also achieves mutual benefits between localization and understanding tasks. Experimental results show that a single GLIPv2 model (all model weights are shared) achieves near SoTA performance on various localization and understanding tasks. The model also shows (1) strong zero-shot and few-shot adaption performance open-vocabulary object detection tasks and (2) superior grounding capability on VI understanding tasks. Code is released caatts::/ggihhi cww. cttmm//mroooofffffGc..coc
Introduction
Recently, general interest arises in building general-purpose vision systems 21 24
56 42], also
called vision foundation models 66 57]. that solve various vision tasks simultaneously, such as image classification [30], object detection [39], and Visual- (VL) understanding B [1] 27]
Of particular interest, is the unification between localization tasks (e.g., object detection 33 and segmentation 8 (0)) and VI. understanding tasks (e.g., VQA 3 and image captioning 11)
two-stage pre- is the common practice in VI community. A long-standing challenge is the unification of localization and understanding, which aims at mutual benefit between these two kinds of tasks, simplified pre-training procedure, and reduced pre-training cost.
Localization pre-training benefits VI. tasks
1
59 and the "localization>>VL
training procedure (4 49 48
32
61
37
However, these two kinds of tasks appear to be dramatically different: localization tasks are vision- only and require fine greingr output (e.g., bounding boxes or pixel masks), while VL understanding tasks emphasize fusion between two modalities and require high-level semantic outputs (e.g., answers or captions).
21 22 56) have made early attempts at unifying these tasks in straightforward multi-task manner, where low-level visual encoder shared across tasks, and two separate high-level branches designed for localization and VL understanding, respectively. The localization tasks still vision-only and do not benefit from the rich semantics vision-language data. As result, such unified models see the marginal mutual benefit or even performance degradation task-specific models.
M compared with
In this paper, we identify "VL grounding" as "meta" ccpaabiityy for localization and understanding capabilities. VL grounding involves not only understanding an input sentence but also localizing
"The two authors contributed equally. Work done at Microsoft Research.
Corresponding author.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Understanding tasks
Phoopi aidaa. ctens saries. Claggis.
21 Grounding
Vous Quesion ArsAsting
InTag Castion
Segmerrationn
Car Umbrella. Sike...
Bik limrrella. Dog...
Agreen umbreits. Aprkcstrced unbretla, plain white unbrolla.
What is gelbodinng (MASS)
pioure (MAAK))
Pronpt parsoe chai:
image input
Eeat Inpuat
Image
Text Encoder
Deep Fusion Block
Where
vacu and
Prodicsonn
bedigrann
GUIPO Unifled Outputs
Unnordal
nan
Annawe unbrells
Figure 1: Left: GLIPv2, pre-trained grounded VL understanding model, unifies various localization and VL understanding tasks. These two kinds of tasks mutually benefit each other, and enables new capabilities such as language-guice detection/seementation and grounded VQA/captioning, Right: Additional examples from ODinW (detection), LVIS (segmentation), VQA, COCO Captioning.
the mentioned entities in the image (see an example in Figure D. We build grounded VL understanding model (GLIPv2) as unified model for localization and VL understanding tasks.
Localization + VL understanding grounded VL understanding. Localization tasks involve both localization and semantic classification, where classification be cast VL understand- ing problem using the ciassification-to-ie to-matching trick (Section 3.1. Therefore, we reformulate localization tasks as VL grounding tasks, which the language input a synthesized sentence as the concatenation of category names (36]. Localization data are turned into VL grounding data, accordingly. The massive VL understanding data (image-text pairs) can be easily turned into VL grounding data in self-training manner [36]. Therefore, GLIPv2 has unified pre-training process: all task data are turned into grounding data and GLIPv2 is pre-trained to perform grounded VL understanding.
stronger VL grounding task: inter-image region-word contrastive learning. GLIP (36 proposes the phrase grounding task as its pre-training task, which we argue is an easy task and does fully utilize data information. For example, in the VL grounding task in Figure [ the phrase grounding task only requires the model to match given image region to one of the three phrases in the text input, i.e., "green, pink striped, or plain white umbrella?". This 1-in-3 choice is very easy, only requires color understanding, but loses lots of information in this grounding data: the umbrellas are not any other colors, like black, yellow, etc; objects in those regions are umbrellas but not any other categories, like car, bike, etc. From a contrastive learning view, this phrase grounding task only has two negatives. More negatives can be created from this annotation and thus enable stronger contrastive learning. In GLIPv2, we introduce the novel inter-image region-word contrastive learning task, which leverages phrases from other sentences in the same batch as potential negatives, as another much stronger VI. grounding task. This new region-word contrastive loss enables GLIPv2 to learn more discriminative region-won word features and demonstrates improvements over all downstream tasks.
GLIPy2 achieves mutual benefit between localization and VL understanding. 1) Experimental results (Table ( show that single GLIPv2 model (all model weights are shared) achieves near SoTA performance on various localization and understanding tasks. 2) Thanks to semantic-rich annotations from the image-text data, GLIPv2 shows superior zero-shot and few-shot transfer learning ability to open-world object detection and instance segmentation tasks, evaluated on the LVIS dataset and the "Object Detection the Wild (ODinW)" benchmark. 3) GLIPv2 enables language-guides detection and segmentation ability, and achieves new SoTA performance on the Plick3OK-entifgg phrase grounding and PhraseCut referring image segmentation tasks. 4) Inherently grounding model, GLIPv2 leads to VL understanding models with strong grounding ability, which are self-explainable

---

and easy to debug. For example, GLIPv2, when GLIPv2 is finetuned on VQA, it can answer questions while localizing mentioned entities (see Figure Tand Section
Related Work
Localization models. Traditionally. localization tasks such as object detection and segmentation
are single-mode modality and output bounding boxes or pixel masks [55 38 23
[4
46
10,
[]] One
challenge of these single-modality models lies in generalization to rare and novel concepts: it is hard to collect localization data that cover many rare categories 200 A long line of research focuses on
this generalization problem, under the name of zero-shot 44, 62 7 63
[T8] 5 [2
or open-vocabulary [58] 19 localization. Built upon MDETR (25] and GLIP [6]. GLIPv2 converts
trick (Section 3). Thus GLIPv2 can learn from the semantic-rich vision-language data and shows strong performance on open-vocabulary localization tasks.
localization tasks into grounded vision-language task using the dlasification-to-matteee
Vision-language understanding models. Vision-language (VL) understanding tasks such as VQA 3. image captioning [M]] and image-text retrieval [26] involve understanding visual se- mantics and how they are expressed in natural language. Many VL models (e-g.. BUTD) 59 rely on pre-trained localization model as their visual encoder; the downside is the pro-longed Plocalization>>VI.. pre-training pipeline 41 49 13 48
34
32 61 37 35 contrast, GLIPv2
simplifies the pre-training pipeline and enables grounded VL understanding for better interpretability (Section 4.4).
made pioneering efforts in unifying localiza- tion and understanding. However, localization tasks are still treated as single-modality tasks, while VL tasks involve two modalities. The unification is achieved via straightforward multi-tasking: low-level visual encoder is shared across tasks and two separate branches are designed for local- ization and VL understanding. Such unified models do not bring evident mutual benefit and often underperform task-specific models. contrast, GLIPv2 identifies grounded VL understanding as a meta-task for localization and understanding. The task unification brings architecture unification: the unified grounded VL understanding model empowers localization branch with VL capacity, arriving unified branch that excels at both tasks.
Unifying localization and understanding. 21 24 56
GLIPv2 GLIP. 1) GLIP shows that grounded pre-training improves localization. GLIPv2 further shows grounded pre- pre-training improves VL understanding and thus leads to a unified model for localization and VL understanding. 2) GLIPv2 introduces the inter-image region- word contrastive loss, which is another and stronger grounding task than the pre-training task in GLIP. The proposed loss can be viewed as region-word level generalization of the prevalent image-level contrastive learning 33 55 3) GLIPy2 outperforms GLIP on all benchmarks with the same pre-training data.
3 GLIPv2: Unifying Localization and VL Understanding
Based on the reformulation of object detection as a generalized phrase grounding task in GLIP [6] we unify both localization and VL understanding tasks as grounded vision-language tasks. A grounded vision-language task takes both image and text as inputs, and outputs region-level understanding results (e.g., detection, segmentation) and/or image-level understanding results with associated grounding/localizatio information (e.g., VQA, image captioning). We will present the unified grounded VI. formulation and architecture in Section 3.1 the pre-training losses in Section 3.2 and transfer to downstream tasks in Section 3.3
3.1 A Unified VL Formulation and Architecture
At the center of GLIPv2's unified formulation is the maasification-tommatciii trick, which refor- mulates any lask-specific fixed-vocab classification problem as an lask-agnostic open-vocabulary vision-language matching problem. The best example is the reformulation of image classification as image-text matching in CLIP 44 which enables the model learn from raw image-text data directly, and achieves strong zero-shot results on open-vocabulary classification tasks. In GLIPv2,
3

---

replace every semantic classification linear layer in traditional single-mon single-modality modality vision models with vision-langenn -aiiun-l matching dot- pro--rrouut layer.
As illustrated in Figure GLIPv2's unified VI. architecture based on the generic architecture we term Architecture II. It consists of dual encoder, denoted Ency and Ener, and fusion encoder, denoted as Enev The model takes an image-texi pair (Img, Text) as input, and extract visual and text features as below:
Ô Ency (Img), P Encz,(Text), O, P Enev y. (O, P),
(I)
where (Ó, P) and (O, P) denote the image/text features before and after VL fusion, respectively.
Vision-Language understanding tasks. Arch II is the most popular model architecture for VL understanding tasks. Given the aross-modality fused representations O and P. it is straightforward to add lightweight task-specific heads for various VL tasks. For example, GLIPv2 adds two-layer MLP on top of text features P' as the masked language modeling (MLM) head, to perform the MLM pre-training. We provide model details of VQA and image captioning in Section 3.3
(Language-guided) object detection and phrase grounding. Following GLIP [3) GLIPv2 uses the paasiffcation-tommachii trick to unify detection and grounding. More specifically, for detection, we simply replace the class logits S.t. = OW", where W is the weight matrix of the box classifier, with task-agnostic region-word similarity logits Sp Syrund -PP", where text features P are label embeddings from task-agnostic language encoder. As shown in Figure T object detection and phrase grounding share the same input/output format and model architecture. See GLIP (36) for more details. Their only difference is the input text format: (1) for object detection, the text input is a string of concatenated candidate object labels; (2) for phrase grounding, the text input is natural language sentence. We refer to GLIP (36 for more details.
(Language-guided) instance segmentation and referring image segmentation. Given the object detection results, an instance segmentation head is added classify each pixel within the box into a semantic class. Again, GLIPv2 uses the classification-to- -onimaation-t-mmcchinnng trick to produce a unified instance segmentation head for the standard instance segmentation tasks and the referring image segmentation tasks and leverage both types of data for its pre-training. This clasifiicttinnto-maathhi also apply to many other semantic classification heads in single modality CV models (e.g., semantic segmentation) and thus transfers them to language-gus -uided CV models.
trick can
3.2 GLIPv2 Pre-training
The GLIPv2 is pre-trained with three pre-training losses: phrase grounding loss Egrondd from a vision- laigunle reformulation of the object detection task, region- word contrastive loss Limer from novel region-word level contrastive learning task, and the standard masked language modeling loss Luum proposed in BERT [16].
CGLIPV2 Live + Lintr Limter + Lual
Gyoun
Similar to losses in detection tasks, the grounding loss feround has two parts: the localization loss Live trains localization heads with bounding- supervision, e.g., RPN loss, box regression loss and/or centerness loss [50]; the intra-image region-wort alignment loss Liura is essentially the semantic classification/rettivaal loss for each region.
Intra-image region-word alignment loss. Given one image-text pair (Img, Text), we obtain the image and text features after cross-modality fusion O and P. The Intra-image region-word alignment loss is computed by
Lintra = (oP".)
where OPT is the similarity score between image regions and word tokens, and is the target affinity matrix determined by the ground-truth annotations. The loss function loss is typically a cross-entropy loss for two-stage detectors [46] and a focal loss [38 for one-stage detectors.
However, as discussed in Section I this intra-image region-word contrastive learning is rather weak in the sense of contrastive learning, due to the limited number of phrases that can one caption can contain. GLIP (36] alleviates this problem by appending few negative sentences to form longer

---

text input with more (negative) phrases, However, constrained by the maximal length of text tokens (256 in GLIP and GLIPv2), only a few negative sentences can be added and the number of negative phrases remains in the order of 10's. This
problem also exists in detection when the input text cannot include all class names in a detection dataset, e.g., Objects365.
data 36
Inter-image region-worr wordd contrastive loss. In GLIPv2, we propose using phrases from other image- text pairs in the same batch as negative examples, which effectively increases the number of negative examples to the order of 1000's, with nearly negligible additional computational cost.
As in ① given batch of image-text pairs (Img', Textt) and their ground-truth annotations (T)) E the model produces the image and text features before and after VI. fusion, denoted as (O),Pi) ı and (O', P*). respectively. Then illustrated in Figure 2 (Left), batch-wise similarity matrix gpound and a batch-wise target affinity matrix Teatch are constructed by considering all the image regions and text phrases across this batch. Their (i,5)'th blocks are obtained as below:
B
Strochal [ = Ö'()iTT Tharchhi [i,5]
(T
if Ti= j
obtained by label propagation, otherwise.
The inter-image region-word contrastive loss is then defined as the standard bi-directional contrastive loss applied on all image regions and phrases in this batch:
Ciner- = cross_ caaiss_ettt_____leett (S ground: 7 Thatth axis 0)-cro entropy cr-osssssenttop canrroy__osssSSt ground Trhttch .xis 1).
(5)
Compared with that in the inter-image contrastive loss (3). the number of negatives is multiplied by batch size B in this inter-image contrastive loss (5) We elaborate two important details in 4 (1)
B after VL fusion, to compute the batch- wise similarity matrix in the inter-image contrastive loss Otherwise, the image and text features after VI. fusion would have seen the paired information ①. and thus the model can easily rule out the negatives from misaligned images/texts. (2) We cannot simply assign all regions and texts from unpaired image-text as negative pairs, as done in the standard contrastive loss in CLIP 44). Instead, we determine the off-diagonal blocks in the target affinity matrix T hhucc by label propagation. For example, as illustrated in Figure 2 (Left), if region is annotated as "person", it should be a positive pair with all "person" " phrases in detection-type texts. We do not propagate positives to grounding-type -type texts (natural sentences) because phrases in sentences carry contexts that are unique to that image-sentence pair.
GLIPv2 uses the image text features (Ò', } before VL fusion, not
Pre-training with both detection and haired-image-tex data. GLIPv2 pre-training data is in the image-text-target triplet format (Img, Text, T), where the target affinity matrix contains the box-label localization annotations. We also use massive image-text pair data (Img, Text) to pre-train GLIPv2, by generating grounding boxes T't for phrases in the text with the GLIP pre-trained model from 33.. The human-annotated ODgrronniig data provides fideliiyy localization supervision, while the massive image-text data greatly improves the concept diversity for GLIPv2.
Second-stage pre-training of the segmentation head. GLIPv2 performs a second-stage pre-training of the Language-guide segmentation head on both instance segmentation and image referring segmentation data, while fixing all other parts of the model.
3.3 Transfer GLIPv2 to Localization and VL Tasks
We introduce two ways to easily transfer GLIPv2 to various downstream tasks. In addition, GLIPv2 can perform conventional VL tasks (e.g.. VQA) along with localization, effectively making every task we consider a "grounded VI. understanding" task.
One model architecture for all. GLIPv2 can be transferred to downstream tasks by fine-tuning tin-ttunin the model with an (optional) task-specific head. 1) For detection and segmentation tasks, no task-specific head is needed as the pre-training architecture can inherently perform detection and segmentation. 2) For VL tasks: for VQA, a classification head is added on top of the hidden representation of the start-of-sequee ofsuuuuncceeeeeeeeeeeeeeeeeeeeee token; for caption generation, which maximizes the likelihood of the next word given context. We use a unidirectional attention mask and prevent the image part from attending to the text in the fusion layers.
train with unidirectional language modeling loss,
One set of weights for all. There a growing interest in developing models that can be transferred to various tasks while only changing the least amount of parameters save training time and storage
5

---

8 6
Lap
Deep Fusion
1
Text
Tott Fronder
PL po
Block
pa
Cuim
Trep Pairs
Propapate
HAF
Pasio Biyyl....
I
I
D
0
O
Iuape Teat?
Cinter: : Inter-image Contrastive Loss
Lincm: Intra Intra-image Contrastive Loss
Figure 2: GLIPv2 pre-training losses: the intra-image alignment loss Liam (right) takes features after VI. fusion and compute loss over region-word pairs within each image-text pair; the inter-image contrastive loss (left) Limer takes features before VL fusion and computes loss over all region-word pairs across batch of image-text pairs. Label propagation is used to determine the off-diagonal blocks of the Lister target matrix
cost 447 311. Following GLIP, GLIPv2 can be transferred to localization tasks in a zero-shot or One single GLIPv2 model can serve various tasks, where
prompt-tuning setting (Section 4.2) each task only keeps few or no parameters. Of particular interest is the prompt tuning setting. For certain localization task, the text prompt is the same for all input images; thus, could directly tune P, small prompt embedding matrix, to adapt GLIPv2 new tasks. Prompt tuning in feep-fused model such as GLIPy2 is different from the conventional linear probing/prompt tuning vision models such as CLIP. The latter can also be viewed as only tuning small prompt/softmax embedding P; however, tuning P only affects the very last layer of the model while the visual representation is still frozen. In contrast, GLIP/GLIPv2's visual representation is conditioned on the prompt embedding P: tuning P changes the text, visual, as well as fused embeddings. As result, prompt tuning in GLIPv2 is highly effective, often matching the performance of fine-tuning (see Table 2). This is in contrast to the common observation in CV that linear probing lags behind fine-tuning
setting [53 60 in shallow-interaction
large gap [22].
Grounded VL understanding. GLIPv2 also enables grounded VL understanding, where retain the ability to perform grounding when fine-tuning the model to a downstream VL task. This increases the interpretability of the model. Specifically, we first turn the VL data of the downstream task into grounded VL data using a pre-trained GLIP model. Then we train the model with both the downstream task head and grounding head. For VQA, the model is trained to predict the answer and ground entities in the question as well as the implied entity in the answer; for captioning, the model is trained to predict the next word given the context and ground the current decoded word. By tuning localization tasks into grounded VL task and augmenting VL tasks with grounding ability, we effectively turn every task into grounded VL. understanding task (see examples in Figure I].
Experiments
In this section, we show that GLIPv2 serves as a performant and easy-to-deploy ceneral-purpose vision system. 1) One Model Architecture for All (Section 4..pp. GLIPv2 can be directly fine-tuned to both localization and VL understanding tasks with minimal architecture change. It achieves performance on par with SOTA models with specialized architectures. 2) One Model Weight for All (Section 4.2) GLIPv2 can be transferred to localization tasks in a zero-shot manner with zero parameter update; with prompt tuning, a single GLIPv2 model can achieve comparable performance with fully fine-tuned settings on both localization and understanding tasks.

---

Type
COCO-Det (test-dev)
ODiaW (teat)
LVIS (minival)
COCO-Mask (test-dev)
Flick30K PhraseCut (test)
VQA (test-dev/ test std)
Cuptioning (Karpthy-esst)
(test)
Mask CNNNN
23
39.8
33.3
3/7.11
DETR
42.0
17.8-
DyHead-T
16
Localization
49.7
60.8
DyHead-L.
15
60.3*
VistaBEET
71.33
70.8/710 73.8/
UNITR
Understandinn
VinVI. 55
76.5 76.6
130.8
GPV 21
62.5 /-
102.3
Localization
42.3
MDETR
Understanding
24.2/-
84.3
53.7
70.6/ 70.6
Unicom 50
80.4
69.2/ 69.4
119.1
GLIPT
36
Localization Understanding
85.7
GLIP-L
36
61.59
68.9
87.1
GL.PPy2TT (Ours)
Localization &
55.5
40.6
53.5/421
86.5
71.671.8
GLIP22-B (Ours)
58.8
69.4
/46.2
59.0/45.8
87.5
61.3
73.17
128.5
GI.IP2-HH (Oues)
Understanding
60.6 (62.4")
70.4
59.8 488.8
59.8 49.9
87.7
61.3
74.67
131.0
Table 1: One model architecture results. For COCO-Det test-dev, * indicates multi-scale evaluation. For LVIS, report the numbers for both bbox and segm minival to avoid data contamination due to the pre- pre-training. For Flickr30K test, we report the metric under R@1. For COCO-Mask, we also report both bbox and segm on test-dev.
Following GLIP 36 we adopt Swin Transformer [40] as the image encoder Ency, text transform- ers [51] 44 as the text encoder Ency, Dynamic Head [1] with language aware deep fusion [36 as the fusion encoder EncyL, I, and Hourglass network 45 as instance segmentation head feature
extractor. We train GLIPv2 at three scales: GLIPv2-T, GLIPv2-B, and GLIPvv-H.
GLIPv2-T has the same model config and initialization GLIP-T: Swin-Tiny and BERT-Bat Base as the dual encoder. The model is pre-trained on the following data: 1) 0365, 2) GoldG as in GLIP-T (C), and 3) Cap4M, 4M image-text pairs collected from the web with boxes generated by GLIP-T 36 GLIPY transformer as dual encoder, and are initialized from the UniCL [55 checkpoints. We observe much stabler training with GPT-type pre-layernorm transformer 17 than BERT-type post-layernorm transformer. The training data contain: 1) FiveODs (2.78M data) ] 2) GoldG as in MDETR (25): and 3) CCI5M+SBU, 16M public image-text data with generated boxes by GLIP-L [6]. Segmentation heads of GLIPv2 models are pre-trained on COCÓ, LVIS 20 and PhraseCut model parameters are frozen.
are based on Wwin-Base/Swin-Huge and the pre-layernom text
with all other
Note All datasets above were collected by the creators (cited) and consent for any personally identifiable information (PII) was ascertained by the authors where necessary. Due to limited space, we refer to supplementary for details of training recipes and hyper-parameters.
4.1 One Model Architecture for All
We compare GLIPv2 to existing object detection and vision-language pre-training methods on a wide range of tasks. We fine-tune the model on different downstream tasks and report the performance in Table I We make the following observations.
GLIPv2 specialized Localization methods. GLIPv2 outperforms previous localization models on generalization to both common and rare classes and domains with a single model archisecture and pre-training stage. OD common categories (COCO-DeC -Dee), GLIPv2-T achieves 5.8 improvement compared to the standard DyHead-T trained on O365 (55.5 v.s. 49.7). GLIPv2-H reaches 62.4 AP on test-dev, and surpass the performance of the previous SoTA model GLIP-L. 2) OD on rare unseen categories (LVIS), GLIPv2-T outperforms a supervised MDETR on the bbox by great margin (59.8 v.s. 24.2). 3) Generalization diverse real-word tasks (ODinw), GLIPv2-T (55.5) performs better than original GLIP-T (64.9) on the average of 13 public datasets; GLIPv2-B outperforms GLIP-L by PhraseCut), for traditional instance segmentation (i.e.. COCO-Mask), GLIPy2-H 2- outperforms the well-known Mask R- R-CNN by great margin on segm.
0.5 AP. 4) Instance segmentation (COCO-Man
Besides 0365, it combines with 4 additional OD datasets including COCO 39 Openlmages Genome 229 and ImageNetBoxes 30
28 Visual
7

---

Direct Evaluation
Prompt Tuning
COCO-Mask
Coco-Der
ODin (tesc)
COCO-Mask (test-dev)
(minival)
(test)
(minival)
(minival)
[Best-dee (eest-dev)
(minival)
(tes)
GLIP-T
4666/
46.5
26,0
85.7
46.5
GLIP-L
49.8/-
52.1
37.3
87.1
58.8
67.9
GLIP22-T
47,3/3577
29,0
86.0
$3.4 can1
64.8
49
34,8
2 carie
49
GLIP22-B
61.9/1344
54.2
48.5
87.2
59.0
67.3
36.8
(35-45)
58.8/44.9
02144
55.9
64.1
(001-17)
Table 2: One set of weights results v.s. Original GLIP. indicates multi- evaluation. Numbers in red clearly points out the difference between the prompt tuning and full fine-tum tne-tuning results (see Table
D. Numbers in gray mean that they are not in zero-shot manner. f: these two numbers are artificially high due to some overlap between COCO-miniva and -isualGenome-train.
F.5 Tuers
Presp Tuap
Model
Zere-Shot
Pronje Tuming / Fine Tusing
3
5
10
All
DyHiend T 56
33.8
43.6
46.4
90.8
60.8
7.1
GUIVT
(GLIP-T
46.5
62.4
Gun Cuar
CyfkecT
Cew + Rens from + fine
55.6
574
66.5
Figure 3: Data efficiency of GLIPv2 on ODin W. The X-axis is the amount of task-specific data, from zero-shot to all data. Y-axis is the average AP across 13 datasets.
Table 3: Zero-shot, prompt tuning, and full fine- tuning performance ODinW. GLIPv2 models exhibit superior data efficiency.
For language-guided segmentation (i.e., PhraseCut), compared to MDETR, GLIPv2-T achieves an improvement of 5.7 mask AP.
GLIPv2 v.s. specialized VL Understanding methods. GLIPv2 rivals with SoTA specialized models for VL tasks. 1) For VQA, GLIPv2 outperforms VisualBERT and UNITER and approaches the previous SoTA model VLn 2) For Captioning. the best GLIPv2 even surpasses VinVL (VinVL and GLIPv2 are not trained with CIDEr optimization).
GLIPv2 v.s. localization and VL models. Prior works such GPV, UniT and Unicom have also explored unifying localization and VL models (see discussion Section 2) GLIPy2 outperforms all previous systems on both localization and VL tasks. For the best GLIPv2-H, it outperforms the UniT by a great margin (18.3 AP) on COCO object detection tasks. Meanwhile, it also surpasses UniT's performance VQA by 6.9 points and GPV's peformance on Image Captioning well.
Takeaway. Most notably, GLIPv2 outperforms previous "unified" models (GPV, UniT, MDETR, Unicorn) by large margin. This is the first time that single model architecture could achieve near SoTA performance on both localization and understanding. In contrast, in prior work, there exists certain trade-off between localization and understanding: models that aim achieve high understanding performance tend to have lower localization performance (e.g., UNIT's detection performance limited to the DETR 9 architecture), it is not trivial merge a SoTA localization branch and a SoTA VL branch into a single model.
4.2 One Set of Model Parameters for All
GLIPy2 is pre-trained to perform grounding: thus can be transferred various localization tasks with changing zero few parameters. We evaluate GLIPv2 under two such settings: 1) direct evaluation, where we transfer the model "as is" without any parameter change, and 2) prompt tuning, where only the prompt embedding is tuned for specific tasks (Section 3.3

---

Direct evaluation. The pre-trained GLIPv2 can be directly evaluated on any object detection task (by concatenating the object categories into a text prompt) and visual grounding task without any further tuning. We evaluate the models on four localization tasks: COCO, ODin LVIS, and Flickr30, and their results are presented in Table 2 Note that for GLIIvv-B and GLIPv2-H, the training sets of Flick30K and LVIS are present in the pre-training data. Thus, reported numbers on these metrics are not zero-shot evaluation (we have marked them gray). For all other evaluation results, the models are evaluated in zero-shot settings without any further tuning.
GLIPv2 can be effortlessly transferred to different localization tasks without further tuning. 1) For COCO, GLIPv2-T achieves a zero-shot performance of 47.3 without seeing any COCO training images. This surpasses well estabbsstalis supervised systems (e.g., Mask R- R-CNN) and also outperforms GLIP-T by 0.7 AP. 2) For ODinW, GLIPv2 also shows strong zero-shot performance. GLIPv2-T (48.5) surpasses the GL.IP-T (46.5). Meanwhile, the zero-shot performance of GI. B and GI.IPv2- H even surpasses the 10-shot tuning performance of DyHead-T (to be introduced in Figure 3 3) For LVIS, GLIPvv-T achieves 3 AP improvement performance compared to the GLIP-T. 4) For Flickr30K, GLIPv2-B achieves even higher number (87.2) compared to original GLIP-L (87.1).
Prompt Tuning. Following GLIP, GLIPv2 supports efficient prompt tuning: the visual representa- tion is heavily conditioned the text representation due the deep fusion block (Section 3.3); thus we could fine-tune only the prompt embedding for each task but still maintain high performance.
Prompt tuning GLiPv2 achieves similar performance as full fine-tuning. When comparing the performance of each task in Table aand at the same time, for GLIPv2, prompt tuning performance almost matches the one model architecture results on localization tasks, without changing any of the grounding model parameters.
4.3 GLIPy2 as a Strong Few-Shot Learner
We demonstrate GLIPv2's performance ODinW datasets with respect to different amounts of training data in Figure 3 The performance improvement between GLIPv2-T and GLIP-T exhibits more superior data efficiency for prompt tuning. We compare with the SoTA detector DyHead-T, pre-trained on Objects365 in Table [ It can be seen that a zero-shot GLIPv2-T (48.5) outperforms a outperforms 5-shot DyHead-T (46.4) while the performance of one-shot GLIPv2-H (61.3) surpasses a all-shot fully supervised DyHead-T (60.8).
4.4 Analysis
Pre-training losses Table [shows the performance of the downstream tasks with different variants of our method. Compared to the GLIP pre-training tasks with only intra-image region-word contrastive loss (Row 3), adding inter-image word-region loss (Row 5) substantially improves the pre-trained model performance across all the object detection tasks (COCO, ODinW, and LVIS) on both zero-shot and fine-tuned manner. Consistent with common observations from most VL understanding methods, adding MLM loss (Row4) benefits for learning the representation for understanding tasks (Flick30k, VQA, and Captioning). Furthermore, using all three losses together at the 1st stage pre-training and doing the 2nd stage pre-training without MLM on OD and GoldG data, GLIPy2 (Row6) can perform well on both the localization and VL understanding tasks.
An additional stage of pre-training is applied for small models (GLIPv2-T and GLIPv2-B) due to limited model capacity. In order to achieve higher performance on both localization and understanding tasks, we find that including all data (even with some noise) and MIM loss in the first stage of pre-training will benefit the model for learning a better representation of both localization and understanding capability. Since the OD tasks require the model with more accurate localization ability, in our 2nd stage of pre-training, we decide to eliminate the MLM loss. The large model (-L does not need this additional stage because it has enough capacity to learn both word- region alignment and MLM together in a single stage.
Pre-training data Table reports the last checkpoint results on GLIPv2 when we do the scaling up of pre-training data. As more weak image-text pair data (Cap) is involved in our training, benefits both bibaard/in-domiin (i.e., COCO, Flickr30K) and large-domain gap (i.e., ODinW, LVIS) tasks. We also show that by adding the inter-image region-word contrastive helps when we are fixing the data at the same scale. For large-domain gap tasks, adding the inter-image region- region-word contrastive

---

Row Model
COCO
ODinW
LVIS Flickr30K VQA
Captioning
No pre-train
-/50.6
-/60.8
64.6
111.5
+ Cait
-/48.5
-/37.4
64.6
110.9
+ Live Linm
46.6/55.2
46.5/64.9
26.0
85.7
69.4
119.7
+ Lvee Linm + Lemim
47.0/55.2
47.6/66.2
28.5
86.5
69.8
120.7
+ Live Lian + Liter
47.1/55.4
48.4/66.3 28.6
85.8
68.7
120.4
Cimn Linor Liim
47.3/55.5
48.5/66.5 29.0
86.3
70.7
122.1
Table 4: Pre-training losses on Tiny-scale model. Involving intra-image region-word alignment loss Lintras inter-image region-word contrastive loss Linter and MLM loss Lmim will benefit both localization and understanding tasks.
Cpsc Pre train Data
coco
ODinW
LVIS
Flck30K
0365, GoldG
0365, GoldG
48.59
48.06
43.14
42.64
25.6
26.9
83,90
84,36
Model
COCO Caption
CIDEr SPICE R@ R@5
Flickr30K Grounding
R@ 10
GoldG, Cap4M
51.35
34.2
0365, GaldG, Capri
48.79
52.70
35.0
85.50
GLIPv2-T
36.5
119.8
21.6
80.8
94.4
96.5
0866, GoldG, Cap12M
48.50
49.32
35.5
85.79
GI.IPv2B 37.4
123.0
21.9
81.0
94.5
96.5
0365, GoldG, Ca
49.26
53.15
36.6
85.84
Table 5: Pre-train data scale up on Base-scale model. Results are reported at the last checkpoint. See supplementary for results at all checkpoints.
Table 6: GLIPv2 can perform captioning and
grounding the same time (a.k.a., grounded VI. understanding).
loss will further boost the model to learn better representation. For more detailed scaling-up up effeets on various tasks under all the checkpoints for GLIP and GI.Iv2, refer to Appendix.
Note that the (Img. Text, T) data used in GLIPv2 pre-training can be just Ktow1&2 in Table with which GLIPv2 pre-training does not involve any pseudo data from pre-trained grounding/locaaization model. In order to achieve the best performance, GLIPv2 uses image-text pair data with pseudo boxes (Cap) from a pre-trained GLIP model (Row3-6 in Table which is trained with the same "grounded VL understanding" task but just with smaller data.
data
Grounded Vision-Language Understanding GLIPv2 can be trained perform a VL task and grounding at the same time (Section [.3). We denote such an ability as grounded VL understanding. In Figure T we showcase grounded predictions of GLIPv2 on VQA and COCO captions. We also conduct quantitative evaluations (Table 6). The model achieves strong performance for both VI. understanding (on COCO Caption) and localization (on Flickr30K Grounding). Such an ability to produce high-level semantic outputs (i.e., answers and captions) and supporting localization results is another appealing trait of GLIPv2, as potential users can have a better understanding of the model behaviour. See more detailed analysis and qualitative examples in the Appendix.
5 Conclusion and Social Impacts
This paper proposes GLIPv2, unified framework for VL represennation learning that serves both localization tasks and VL understanding tasks. We experimentally verify the effectiveness of the unified model and the novel region-word contrastive learning. Compared to existing methods, GLIPv2 achieves competitive near SoTA performance on various localization and understanding tasks. However, additional analysis of the data and the model is necessary before deploying it in practice since large-scale web data may contain unintended private information, unsuitable images/text, or some bias leakage. Further investigation may be needed for web data due to the above issues.
6 Acknowledgement
We thank anonymous reviewers for their comments and suggestions. Additional thanks go to the Microsoft Research Horizontal AI Team and Microsoft Alexander Multi-modal Team for providing computer resources for large-scale training. The baseline models used in our experiments are based on the open-source code released in the GitHub repository; we acknowledge all the authors who made their code public, which tremendously accelerates project progress.
10

---

References
[1] Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6077-6086. IEEE (2018)
[2] Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., Gould, S., Zhang, L.: Bottom-up and top-down attention for image captioning and visual question answering. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 6077-6086 (2018)
[3] Antol, S., Agrawal, A., Lu, Mitchell, M., Batra, D., Zitnick, C.L.,, Parikh, D.: VQA: Visual Question Answering. In: International Conference on Computer Vision (ICCV) (2015)
[4] Bansal, A., Sikka, K., Sharma, G., Chellappa, R., Divakaran, A.: Zero-shot object detection. In: Proceedings of the European Conference Computer Vision (ECCV). pp. 404 (2018)
[5] Bilen, H., Vedaldi, A.: Weakly supervised deep detection networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2846-2854 (2016)
[6] Bommasani, R., Hudson, D.A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M.S., Bohg. J., Bosselut, A., Brunskill, E., et al.: On the opportunities and risks of foundation models. arXiv preprint 2xiv:2108.0722 (2021)
[7] Bucher, M., Vu, T.H., Cord, M.. Pérez, P.: Zero-shot semantic segmentation. Advances in Neural Information Processing Systems 32 (2019)
[8] Caesar, H., Uijlings, J., Ferrari, V.: Coco-stuff: Thing and stuff classes in context. In: Pro- ceedings of the IEEE conference on computer vision and pattern recognition, pp. 1209-1218 (2018)
[9] Carion, N., Massa, F., Synnaeve, G., Usumier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: European Conference on Computer Vision, pp. 213-229. Springer (2020)
[10] Chen, K., Pang. J., Wang, J., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Shi, J., Ouyang,
W., et al.: Hybrid task cascade for instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4974 4983 (2019)
[11] Chen, X., Fang, H., Lin, T.Y., Vedantam, R., Gupta, S., Dollár, P., Zitnick, C.L.: Microsoft COCO captions: Data collection and evaluation server. arXiv preprint arXiv:1504.003: (2015)
[12] Chen, Y.C.. Li. L., Yu. L., El Kholy, A., Ahmed, F., Gan, Z.. Cheng, Y., Liu, J.: UNITER:
Universal image-text representation learning. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 104-120. Springer (2020)
[13] Chen, Y.C., Li, L., Yu, L., Kholy, A.E., Ahmed, F., Gan, Z., Cheng, Y., Liu, J.: Uniter: Learning universal image-text representations. arXiv preprint arXiv:1909 1MN::1999.11770 11740 (2019)
[14] Dai, J., Li, Y., He, K., Sun, J.: R-fen: Object detection via region-basee based fully convolutional networks. In: Advances in neural information processing systems. pp. 379-387 (2016)
[15] Dai, X., Chen, Y., Xiao, B., Chen, D., Liu, M., Yuan, L.., Zhang, L.: Dynamic head: Unifying object detection heads with attentions. In: Proceedings of the IERF/CVF Conference on Computer Vision and Pattern Recognition. pp. 7373-7382 (2021)
[16] Devlin, J., Chang, M.W., Lee, K., Toutanova, K.: Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv: 10.04805 (2018)
[17] Gao, P., Geng, S., Zhang, R., Ma, T., Fang, R., Zhang, Y., Li, H., Qiao, Y:: Clip-adapter: Better (2021) [18] Gokberk Cinbis, R., Verbeek, J., Schmid, C.: Multi-fold mil training for weakly supervised object localization. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 2409-2416 (2014)
vision-language models with feature adapters. arXiv preprint arXiv:21
[19] Gu, X., Lin, T.Y., Kuo, W., Cui, Y.: Zero-shot detection via vision and language knowledge distillation. arXiv preprint arXiv:2104.1388
(2021)
[20] Gupta, A., Dollar, P., Girshick, R.: Lvis: A dataset for large vocabulary instance segmentation. In: Proceedings the IREE/CVF Conference on Computer Vision and Pattern Recognition. pp- 5356-5364 (2019)

---

[21] Gupta, T., Kamath. A., Kembhavi, A., Hoiem, D.: Towards general purpose vision systems.
arXiv preprint arXiv:2104.00743 (2021)
[22] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.: Masked autoencoders are scalable vision learners. arXiv preprint arXiv:2111.0633 (2021)
[23] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE
international conference on computer vision. pp. 2961-2969 (2017)
[24] Hu, R., Singh, A.: Unit: Multimodal multitask learning with a unified transformer, In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision. pp. 1439-1449 (2021)
[25] Kamath, A., Singh, M., LeCun, Y., Synnaeve, G., Misra, I,, Carion, N.: Mdetr-modulated detec- tion for end-to-end multi-modal understanding. In: Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 1780-1790 (2021)
[26] Karpathy, A., Joulin, A., Fei-Fei, L.F.: Deep fragment embeddings for bidirectional image sentence mapping. Advances in neural information processing systems 27 (2014)
[27] Kiros, R., Salakhutdinov, R., Zemel, R.S.: Unifying visual-semantic embeddings with multi- modal neural language models. arXiv preprint arXiv: 1411.2539 (2014)
[28] Krasin, I., Duerig, T., Alldrin, N., Ferrari, V., Abu-EI-Haija, S., Kuznetsova, A., Rom, H., Uijlings, J., Popov, S., Veit, A., et al.: Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://github com/openimages 2(3), 18 (2017)
[29] Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, L.J., Shamma, D.A., al.: Visual Genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (JCV) 123(1), 32-73 (2017)
[30] Krizhevsky, A., Sutskever, Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems 25, 1097-1105 (2012)
[31] Lester, B., Al-Rfou, R., Constant, N.: The power of scale for parameter-efficient prompt tuning.
arXiv preprint arXiv:2104 100222444000999 (2021)
[32] Li, G., Duan, N., Fang. Y., Jiang, D., Zhou, M.: Unicoder- VL: A universal encoder for vision
and language by cross-modal pre-training. arXiv preprint arXiv: www::1908.0000 00066666 (2019)
[33] Li, Selvaraju, R., Gotmare, A., Joty, S., Xiong, C., Hoi, S.C.H.; Align before fuse: Vision and language representation learning with momentum distillation. Advances Neural Information Processing Systems 34 (2021)
[34] Li, L.H., Yatskar, M., Yin, D., Hsich, C.J., Chang, K.W.: Visualbert: A simple and performant baseline for vision and language. arXiv preprint 11tiv:198.03300. (2019)
[35] Li, I.H., You, H., Wang, Z., Zareian, A., Chang, S.F., Chang, K.W.: Unsupervised vision-and- language pre-training without parallel images and captions. arXiv preprint arXiv:2010.12831 (2020)
[36] Li, L.H., Zhang, P., Zhang, H., Yang, J., Li, C.,, Zhong, Y., Wang, I.., Yuan, I.., Zhang, I.,, Hwang, J.N., et al.: Grounded language-image pre-training. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 10965-10975 (2022)
[37] Li, Yin, X., Li, C., Zhang, P., Hu, X., Zhang, I.., Wang, I.., Hu, H., Dong, I.,, Wei, F., et al.: Oscar: Object-semantics aligned pre-training for vision-language tasks. In: Proceedings of the European Conference on Computer Vision (ECCV). pp. 121-137. 121-i33.Springer (2020)
[38] Lin. T.Y.. Goyal, P., Girshick, R., He, K.. Dollár, P.: Focal loss for dense object detection. In: Proceedings of the IEEE international conference on computer vision. pp. 2980-2988 (2017)
[39] Lin, T.Y., Maire, M.. Belongie, S., Hays, J., Perona, P., Ramanan, D.,, Dollár, P., Zitnick, C.L.:
Microsoft coco: Common objects in context. In: European conference on computer vision. pp. 740-755. Springer (2014)
[40] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. arXiv preprint arXiv:2103. 14030 (2021)
[41] Lu, J., Batra, D., Parikh, D., Lee, S.: ViLBERT: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In: Advances in Neural Information Processing Systems (NcurIPS), pp. 13-23 (2019)
12

---

[42] Lu, J., Clark, C., Zellers, R., Mottaghi, R., Kembhavi, A.: Unified-io: A unified model for vision, language, and multi-modaa tasks, arXiv preprint arXiv:2206.089) (2022)
[43] Newell, A., Yang. K., Deng, J.: Stacked hourglass networks for human pose estimation. In: European conference on computer vision. pp. 483-499. Springer (2016)
[44] Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell,
A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language
supervision. In: International Conference on Machine Learning (ICML) (2021)
[45] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE conference on computer vision and pattern recognition.
pp. 779-788 (2016)
[46] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems 28, 91-99 (2015)
[47] Shin, T., Razeghi, Y., Logan IV, R.L., Wallace, E., Singh, S.: Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv.2010.15980 (2020)
[48] Su, W., Zhu, X., Cao, Y., Li, B., Lu, L., Wei, F., Dai, J.: VL-BERT: Pre-training of generic visual lingulling representations. arXiv preprint arXiv: I 908. 00530 (2019)
[49] Tan, H., Bansal, M.: Lxmert: Learning cross-modality encoder representations from transform- ers. arXiv preprint arXiv:1908 07490 (2019)
[50] Tian, Z., Shen, C., Chen, H. He, T.: Fcos: Fully convolutional one-stage object detection. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9627-9636 (2019)
[51] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I: Attention all you need. In: Advances in neural information processing systems. pp. 5998-6008 (2017)
[52] Wang, P., Cai, Z., Yang, H., Swaminathan, G., Vasconcelos, N., Schiele, B., Soatto, S.: Omni- detr: Omni-supervised object detection with transformers. arXiv preprint arXiv:2203.10089 (2022)
[53] Wang, X., Huang, T.E.,, Darrell, T., Gonzalez, J.E., Yu, F: Frustratingly simple few-shot object detection. arXiv preprint arXiv:2003.0699 (2020)
[54] Wu, C., Lin, Z., Cohen, S., Bui, T., Maji, S.: Phrasecut: Language-based image segmentation
in the wild. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 10216- 10225 (2020)
[55] Yang, J., Li, C., Zhang, P., Xiao, B., Liu, C., Yuan, L., Gao. J:: Unified contrastive learning in image-text-label space. arXiv preprint arXiv:2204.036 (2022)
[56) Yang, 7., Gan, Z., Wang, J., Hu, X., Ahmed, F., Liu, 7., Lu, Y., Wang, L.: Crossing the format boundary of text and boxes: Towards unified vision-language modeling. arXiv preprint
arXiv:2111.1208: (2021)
[57] Yuan, I.., Chen, D., Chen, Y.I.,, Codella, N., Dai, X., Gao, J., Hu, H., Huang, X., Li, B., Li, C., et al.: Florence: new foundation model for computer vision. arXiv preprint arXiv:2111.11432 (2021)
[58) Zareian, A., Rosa, K.D., Hu, D.H., Chang, S.F.: Open-vocabulary object detection using captions. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 14393-14402 (2021)
[59] Zhang, P., Li, X., Hu, X., Yang. J., Zhang, L.., Wang, L.,, Choi, Y., Gao, J.: Vinvi: Revisiting visual representations in vision-language models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5579-5588 (2021)
[60] Zhou, K., Yang. J., Loy, C.C., Liu, Z.: Learning to prompt for vision- models. arXiv (2021)
preprint (1w.::::199001110
[61] Zhou, L., Palangi, H. Zhang, L., Hu, H., Corso, JJJ., Gao, J.: Unified vision-language pre- training for image captioning and VQA. AAAI (2020)
[62] Zhu, P., Wang, H., Saligrama, V.: Zero shot detection. TEEE Transactions on Circuits and Systems for Video Technology 30(4), 998-1010 (2019)
13

---

[63] Zhu, P., Wang, H., Saligrama, V. Don't even detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11693-11700 (2020)
even look once: Synthesizing features for zero-shot
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] to [Yes], [No], , or [N/A] You are strongly encouraged to include justification your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes]
Did you include the license to the code and datasets? [No] proprietary.
The code and the data are
Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [Yes]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- URL)? [Yes]
mental results (either in the supplemental material hyperparameters, how they
(b) Did you specify all the training details (e.g., data splits,
were chosen)? [Yes]
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent obtained from people whose data you's 're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "cTZCcEuHSIE", "reviewer": "Reviewer_WQgW", "summary": "This paper presents a new V&L model, GLIPv2, that builds upon the original GLIP v1 model centered around multimodal pretraining. GLIPv2 aims to unify visual localization tasks (e.g. object detection) and uses V&L understanding tasks like visual question answering. To do so, three distinct pretraining tasks are used 1) phrase grounding, where the model computes the alignment between image regions and tokens 2) and 3) the standard masked language modeling task. Notably, task-specific classification heads are not used for pretraining. In finetuning on downstream tasks, GLIPv2 performs competitively. The model can also be used in zero-shot and prompt-tuned settings.", "questions": "*Questions*\n\n- Mentioned in the previous section as well, is the text transformer from a pretrained CLIP model (L239)?\n- What are the authors thoughts about requiring another large, multimodal model to generate pretraining data?\n\n*Suggestions*\n\n- About half of the prior work section provides details about GLIPv2; space can be saved by compressing some of the details of this paper. For instance L75-78, which is about half of the localization models subsection, describe the novel contributions of this work instead of prior work and how GLIPv2 differs.\n- This sentence (“See GLIP (36) for more 132 details”) is essentially repeated twice in Section 3.1; one can be removed to save space and for readability.\n\n*Typos*\n\n- L103: output -> outputs\n- caption Figure 2: compute -> computes\n- L118: extract -> extracts\n- L128: classier -> classifier\n- L139: tyeps -> types\n- L206: one set of weight -> one set of weights\n- L207: task -> tasks\n- L210: keep -> keeps\n- L229: broken link to figure\n", "limitations": "- Section 5 is titled “Conclusion and Social Impacts” without a description of social impacts.\n\n- Any biases learned by GLIP (v1) – for instance, having significantly lower detection accuracy for certain demographics of people in images or performing worse for images where people do not fit in the gendered roles of the training data – propagates to the pretraining data for GLIPv2 as well. This is compared to detection datasets that were hand-annotated. It’s a component of using large-scale data, and particularly using large scale data labeled by another ML model, that the authors should address this in their limitations section.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- Similar to CLIP, GLIPv2 can perform open-vocabulary tasks because of the classification-to-matching trick that computes the dot product between the fused visual and linguistic representations. This means the model is particularly adaptable compared to many other V&L models because it can handle new and out-of-domain visual classes.\n- The zero-shot and prompt-tuning experiments have impressive results and are, in and of themselves, exciting to see as tasks for a large V&L model. The ability to evaluate models on downstream tasks with either little or no parameters updates and remain comparable to full finetuning shows the model has learned a lot during pretraining.\n\n**Weaknesses**\n\n- GLIPv2 uses GLIP (v1) to generate bounding boxes for the unlabeled (image, text) pairs in the pretraining data versus just an off-the-shelf object detector that has not had linguistic supervision. This also clouds the data and pretraining approach a bit given how similar GLIPv2 is to GLIPv1.\n\n- The text transformer appears to use the text transformer from CLIP including its pretrained weights (let me know if I'm misunderstanding). If it does use those weights, this skews these results because CLIP's text transformer already had visual supervision from the contrastive pretraining.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ll2znDYLTLm", "reviewer": "Reviewer_GxcW", "summary": "This paper proposes a GLIPv2 model trained on vision-language grounding, where localization and vision-language understanding tasks are reframed through the lens of grounding to have a unified model. Additionally, a new inter image-text token loss is introduced which provides some performance gains. \n", "questions": "Questions\n- On L82-84 authors write “Many VL models (e.g., BUTD) (2; 58) rely on a pre-trained localization model as their visual encoder; the downside is the pro-longed “localization->VLP” pre-training pipeline (41; 48; 13; 47; 34; 32; 60; 37; 35). In contrast, GLIPv2 simplifies the pre-training pipeline and enables grounded VL understanding for better interpretability.\" Doesn’t GLIPv2 use image tags/bbox labels from a pre-trained detector too? Again, paper states “ triplet format (Img, Text, T)” data inputs where T are (box-label) annotations, so don’t we still start with localization to some extent?\n- L183-185 “We do not propagate positives to grounding-type texts (natural sentences) because phrases in sentences carry contexts that\n185 are unique to that image-sentence pair.” Out of curiosity did you try to include these as positives? If so, what happened? Seems like a thoughtful design choice.\n\nStyle and writing comments\n- The writing grammar is not consistently correct throughout the paper.\n- L56 “world” → word\n- L78 bold section should probably be on a new line. Same with L93?\n- L93: arriving *at* a…\n- L128: “classier” → “classifier”\n- L179: “easy” → “easily”\n- L229: Figure reference is missing.\n- L206: One set of weight*s*, L207 task*s*", "limitations": "The authors did not include limitations or negative societal impact discussion. ", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The losses and unified framework introduced in the paper are sound and an important direction for vision-language work. It is ideal to have a single model capable of performing both grounding and understanding type tasks, and making the vision-language understanding tasks more grounded for interpretability and improved performance.\n- The figures are helpful, well made illustrations and there are extensive experiments for the paper’s method to be validated.\n- The method does outperform its predecessor GLIP consistently, albeit small gains. But it does have large performance improvements on VL understanding tasks compared to prior work MDETR and others (e.g., PhraseCut, VQA)\n\nWeaknesses\n- While there are some performance improvements over GLIP, I don’t see significant method changes other than the inter loss. This loss does not seem to contribute that much (1-2 point improvement compared to the ablations without the loss), and the performance improvement compared to GLIP is quite small.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "5ElAx_bDHgL", "reviewer": "Reviewer_iKXy", "summary": "This work proposes a new VL grounding framework called GLIPv2, which unifies several localization and VL understanding tasks in a same unified interface. It shows that doing pretraining on localization data + image-text pairs with this setup improves downstream model performance on all taska and achieves SOTA performance on most of them. It inroduces an inter sample contrastive loss which improves performance. ", "questions": "- Table 4, Row 6 .. is the MLM objective trained alongside other objectives? Or is there another step of pretraining here? Appendix section 4 mentions that Row 6 has additional stage of training without MLM. I think it will be beneficial to add more analysis behind the intuition of a second stage of pretraining.\n- How does the training time gets impacted with the newly introduced loss? \n- Since the authors discuss how the loss differs from CLIP, did they also compare on vision classification tasks?", "limitations": "Yes they have but I would encourage them to discuss in detail any potential negative societal impact of their work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- Proposed work improves SOTA performance on a variety of localization and vision-language understanding tasks\n- The paper is well written and structured\n- They present both finetuning with individual task specific heads as well as prompt tuning and using same weights for different tasks for zero shot\n- The authors do thorough ablation on all the different combinations of the pretraining objectives and dataset combinations\n\n\n\n### Weaknesses\n- I am concerned about the novelty as the additional loss term is somewhat similar to well known region-word loss applied over full batch in multiple works and same setup as GLIP but showing performance on new vision+language tasks. Although this work shows better performance compared to previous works, I wonder if this is also due to additional data used for pretraining which is the localized data generated by GLIP. \n- It seems there is additional localization data generated using GLIP model that is used for training. If so, are the GLIP and GLIPv2 models trained in a teacher/student setup? In fact, how does a GLIP model perform if retrained on this additional data (kind of like self-training)?  Are the GLIP baselines in Table 1 using this additional localization data? If not, then the comparison is a bit unfair.\n- Line 1184-185 is unclear to me what the authors are trying to say. I understand the explanation why Inter-image region-word loss is different from CLIP but the methodology is not very clear.\n- Table 2 if there is trian-test overlap, I would suggest the authors to remove those results. Authors should have been careful about this while preparing the training dataset to deduplicate the train set against any downstream test-validation sets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept