Fine-Grained Semantically Aligned Vision-Language Pre-Training
Authors: Juncheng Li
XIN HE
Longhui Wei
Long Qian
Linchao Zhu
Lingxi Xie
Yueting Zhuang
Qi Tian
Siliang Tang
Venue: NeurIPS 2022 Accept (2022)
Keywords: Vision-Language Pre-Training
Multimodal Pre-Training
Vision and Language
Cross-Modal Reasoning
Image-Text Retrieval

Abstract: Large-scale vision-language pre-training has shown impressive advances in a wide range of downstream tasks. Existing methods mainly model the cross-modal alignment by the similarity of the global representations of images and text, or advanced cross-modal attention upon image and text features. However, they fail to explicitly learn the fine-grained semantic alignment between visual regions and textual phrases, as only global image-text alignment information is available. In this paper, we introduce LOUPE, a fine-grained semantically aLigned visiOn-langUage PrE-training framework, which learns fine-grained semantic alignment from the novel perspective of game-theoretic interactions. To efficiently estimate the game-theoretic interactions, we further propose an uncertainty-aware neural Shapley interaction learning module. Experiments show that LOUPE achieves state-of-the-art performance on a variety of  vision-language tasks. Without any object-level human annotations and fine-tuning, LOUPE achieves competitive performance on object detection and visual grounding. More importantly, LOUPE opens a new promising direction of learning fine-grained semantics from large-scale raw image-text pairs.

Fine-Grained Semantically Aligned Vision-Language Pre- Pre-Training
Juncheng Li 1.2+
Xin He
Longhui Wei 2t
Long Qian
Linchao Zhu
Lingxi Xie
Yucting Zhuang It
Qi Tian 24
Siliang Tang i:
Zhejiang University, Huawei Cloud
fjucheengli, gianlong0926, zhulinchas yzhuang, si siliangl @z ezju.edde edu (hexin80, weilonghuil, tian.qil/ (hhuawei. com, 198908xc0gmail com
Abstract
Large-scale vision-language pre-training has shown impressive advances a wide range of downstream tasks. Existing methods mainly model the cross-modal alignment by the similarity of the global representations of images and texts, or advanced modal attention upon image and text features. However, they fail to explicitly learn the fine- ane-grained semantic alignment between visual regions and textual phrases, only global image-text alignment information available, In this paper, we introduce LOUPE P, fine-grained semantically aLigned visiOn- langUage PrE-training framework, which learns fine-grained semantic alignment from the novel perspective of game-theore theerthe interactions. To efficiently compute the game-theoretic interactions, further propose an uncertainty-ave aware neural Shapley interaction learning module. Experiments show that LOUPE achieves state- of-the-art performance on a variety vision- laiiuue tasks. Furthermore, without any object-level human annotations and fine-tuning, LOUPE achieves competitive performance on object detection and visual grounding. More importantly, LOUPE opens promising direction of learning fine- grained semantics from large- scale raw image-text pairs. The repository of this work is at hhtpp:: / com/ YYJMJC/LOUPP
Introduction
Learning transferable cross-modal representations from large-scale vision-language pre-training has exhibited remarkable performance wide variety of downstream tasks. Most existing works be classified into two categories: dual-encoder and ression-nncode.. The dual-encoder methods [17] 26
35 (49 adopt two separate encoders embed images and texts, and model the cross-modal alignment by the cosine similarity between the global features of images and texts. While such architecture efficient for large-scale image-text retrieval by pre-computing image and text representations offline, they fail to model fine-grained semantic alignment between visual regions and textual phrases. On the other hand, the fusion-encoder methods 7
19
20
25
30
34
43
42 attempt to use
single multi-modal encoder to jointly model the concatenated sequence of images and texts. These methods simulate soft alignment via advanced cross-modal attention
(45) However, they can only leam implicit alignment by end-to-end training, lacking explicit supervision to encourage semantic
Work done when interning at Huawei Cloud.
EEuul Contribution.
*Coresponding Authors.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

alignment between visual regions and textual phrases. And the learned cross-modal attention matrices are often scattering and uninterpretable. Further, they are inefficient for retrieval since it requires jointly encoding every image-text pair during inference.
Learning fine-grained semantic alignment from image-text pre-training is crucial many cross- modal reasoning tasks (c.g., visual grounding [51]. image captioning [48)), but it is particularly challenging as the alignment information between visual regions and textual phrases is not available, posing fine-grained semantic alignment learning a weakly-supervised learning problem. In this paper, we address this problem while simultaneously maintaining high retrieval efficiency by proposing LOUPE fine-grained semantically aLLined visiOn-lange Uage PrE-training framework, from the novel perspective of game theory. We formulate imput patch and word tokens as multiple players into a cooperative game and quantify theerete interactions (i.e., Shapley Interaction [12 40) among them to investigate the semantic aligament information. LOUPE learns fine-grained semantic alignment from two stages: voken-level, Shapley interaction modeling and semantics-leve Shapley interaction modeling, where we first learn to identify semantic regions of images that correspond to some semantically meaningful entities, and then align these regions with phrases in the paired text.
Specifically, token-level Shapley interaction modeling aims to group patch tokens of images into semantic regions that semantically correspond to some visual instances. From the game-theoretic view, we take patch tokens as players and the similarity score between images and texts as the game function. Intuitively, supposing a set of patch tokens correspond to a visual instance in the image, then they tend to have strong interaction to form the complete semantics of the corresponding instance, which contributes to the better similarity judgment with the paired text. Based on this insight, we take the token-level Shapley interaction as soft supervision labels to encourage the model to capture semantic regions from images. Then, semantics-les Shapley interaction modeling infers the fine-grained semantic alignment between semantic regions and phrases. We consider every region and phrase as players and define a fine-grained similarity score as the game function. If region and phrase have strong correspondence, they tend to interact with each other and contribute to the fine-grained similarity score. By measuring the Shapley interaction between each region-phrase pair, we obtain the alignment information to guide the pre-training model.
As computing the exact Shapley interaction is an NP-hard problem 32, existing methods mainly employ sampling-basee method 6 to obtain unbiased estimation. However, as the number of players grows, they require thousands of model evaluations. To reduce the computational cost, we further propose an efficient hybrid Shapley interaction learning strategy, where an uncertainty-awe Shapley interaction learning module cooperates with the sampling-base -baiig method. Experimental results show that our hybrid strategy significantly saves the computational cost while maintaining the estimation accuracy. More analysis is shown in Section 4.5
neural
Our framework serves as a proxy training objective that explicitly establishes the fine-grained semantic aligmment between local region and phrase representations. This proxy objective can be directly removed for downstream tasks, rendering an efficient and semantics-sensiiiee Experiments show that LOUPE achieves new state-of-the-art on image-text retrieval benchmarks. For text-to-image retrieval on MSCOCO, LOUPE surpasses its strongest competitor by 4.2% on recall@ 1. Further, without any fine-tuning, LOUPE successfully transfers to object detection and visual grounding tasks in zero-shot manner. For object detection, it achieves 12.1% % mAP on COCO and 19. 99.5% mAP on PASCAL VOC. For visual grounding, it achieves 26.8% accuracy on RefCOCO and 23.6% accuracy on RefCOCO+. Our contributions are summarized as follows:
dual-encoder model.
We propose LOUPE, , that explicitly learns fine- greinrd semantic alignment between visual regions and textual phrases while preserving the high retrieval efficiency of dual-encoder.
We introduce an efficient and effective hybrid Shapley interaction learning strategy, based on an uncertainty-aware neural Shapley interaction learning module and a sampling- based method.
Pre-trained on image-text data, LOUPH achieves new sate--oftth-arr on image-text retrieval and successfully transfers to the tasks that require more fine- -raggrriined object-level visual under- standing (i.c., object detection and visual grounding) without any fine-tuning.
As manual annotations for masses of object categories is time-consuming and unscalable, our work demonstrates a promising alternative, that is, learning fine- mne-grai semantics from raw texts about images, which are casily available and contain a broader set of visual concepts.

---

2 Related Work
Vision-Language Pre- Pre-Training. The great success of pre-train-and-fie--e tuneeen---uuee paradigm in natural language processing [5] and computer vision [10 14 47 has been expanded to the joint domain of vision and language 12 22) Dominant vision-language pre-training models can be categorized
into two groups: dual-encoder and fission-encoder. The dual-encoder methods 17 26
K5 49] adopt
two individual encoders to embed images and texts separately, and model the cross- modal interaction by cosine similarity. Such architecture is efficient for large-scale image-text retrieval image and text representations can be pre-comme computed offline. However, simply measuring the cosine similarity between global representations is shallow to capture fine-raained semantic relationships between 30
regions and phrases. The fusion-encoder methods 7 13 16,
19
20 24
34
50 55
adopt single multi-modal encoder to jointly model the concatenated sequence of images and texts, which achieves deeper cross-modality interaction. However, these methods are less efficient as images and texts are intertwined compute the cross-modal attention and can be pre-computed offline. Further, there are no explicit supervision signals to encourage the alignment between regions and [56] attempt leverage an off-the-shelf object detector to extract object features for pre- are-training. However, the detector is usually pre- are-trained on limited object categories. Furthermore, considering the excessive demand on memory and computation. existing methods usually fix the parameters of detection models and regard region detection as pre-processing step, disconnected with vision-language pre-training. Thus, the performance is also 40 uses token-wise maximum similarity to enhance the modll interaction of dual-encoder methods. To learn explicit fine- ene-grained semantic alignment, GLIP [23) and X- VLM (33) utilize human-annotate datasets, where regions with bounding-box annotations are aligned with text descriptions. Such a manner is time-consuming and hard to scale to larger raw image-text data from the Internet. In contrast, our proposed framework explicitly learns the fine-grained semantic alignment from raw image-text data and at the same time maintains the high efficiency of dual-encoder. Detailed diseussions can be found in Appendix K.
phrases. Some works 7 24 25
30
43
50
55
restricted by the quality of detection models. FILIP
Shapley Values. The Shapley value [40] was initially introduced in game theory. has been theoretically proven to be the unique metric to fairly estimate the contribution of each player in a cooperative game such that certain desirable axioms are satisfied [46] With solid theoretic foundations, Shapley value has recently been studied as post-hoc explanation methods for Deep Neural Networks (DNN) on Shapley value to interpret the predictions of DNN, Ren er al. (8 propose to explain adversarial attacks by Shapley value. this paper, we propose to model fine-grained semantic alignment by game-theoretic interactions, along with an efficient Shapley interaction learning strategy.
Lundberg et al. [1 propose a unified attribution method based
3 Method
In this section, we first introduce the problem formulation of fine-grained semantically aligned vision-language pre-training Section Then, we propose the corresponding LOUPE framework for fine-grained semantic alignment learning in Section 3.2 and an efficient approach for Shapley interaction learning in Section 3.3
3.1 Problem Formulation and Model Overview
Generally, vision-language pre-training aims to Iearn an image encoder fi and text encoder fr by cross-modal contrastive learning, where the matched image-text pairs are optimized to get closer and the mismatched pairs are optimized to get further. Let fi(T+) and (T;), denote the global representations of the image and text. Then the cross-modal contrastive loss can be formulated as:
exp(fi(L)T (TT)//)) log exp(si(!)) (TTT)/T))
LCMC log Σ exp(((()) (T(TT)/7) ΣS exp(((()) (T(3//T)
(I)
where B is the batch size and T is the temperature hyper- parameter.
While intuitive, such manner can only learn coarse alignment between images and texts but fails to explicitly capture the fine-grained semantic alignment between visual regions and textual phrases. To leamn fine- finengrained semantic alignment while simultancously maintaining high retrieval efficiency,
3

---

Phrase-Regio Reasee
Semantics-
Semantic Alignment
Shapley Interaction
Semantic Phrases (h))
Sementic Regions
(b/)
AFSA
A in blue costis looking down at dog led front of church
LTA
Token-level
Semantic Region Generation
Shapley Interaction
(CLS)
Text-Image Contrastive Learning
[CLS]
Word Tokaass (x])
Patch Tokens [x
LMC
Encoder
Image Encoder
Embeddings
Embeddings
A girl in a blue cost is looking down at dog led front church
Efficient and Semantics-Sensillee Eucoder
Figure I: Overview of LOUPE. Our framework serves as proxy training objective that encourages the image encoder capture semantic regions and establishes the semantic alignment between region and phrase representations. The proxy training objective can be easily removed for downstream tasks, rendering an efficient and semantics-sensitive dual-encoder.
propose LOUPE, fine-grained semantically aligned vision- lision-language pre-training framework that germinates from cooperative game theory.
As illustrated in Figure [ LOUPE learns fine-grained semantic alignment from two stages: token- level Shapley interaction modeling and semantics-level Shapley interaction modeling. For token-level Shapley interaction modeling, we learn aggregate patch tokens of images into semantic regions that semantically correspond to some visual concepts, under the guidance of token-based semantic aggregation loss CTSA. As for semantics-level Shapley interaction modeling, the semantic alignment between the aggregated regions and textual phrases is learned, supervised by the fine- grained semantic alignment loss LFSA. Combined with the two newly proposed losses, the full objective of fine-grained semantically aligned vision-languag pre-training can be formulated as:
C = LCMc + LISA + LESA
(2)
Such a new pre-training objective enforces the image encoder to capture semantic regions and establishes fine-grained semantic alignment between visual regions and textual phrases. During inference, it can directly removed, rendering an efficient and semantics-sensitive dual-encoder.
3.2 Interpreting Fine-Grained Semantic Alignment as Game-Theoretic Interaction
3.2.1 Preliminaries
Shapley Values. The Shapley value 40 is classic game theory solution for the unbiased estimation of the importance or contribution each player cooperative game. Considering a game with W {1, n players, S Ç N denotes a potential subset of players. A game vt-) is implemented as function that maps each subset S of players to score, modeling the outcome of a game when players in S participate in. Specifically, v(N) v(z) denotes the contribution obtained by all players the game. The Shapley value (iilN) for player i defined the average marginal contribution of player i to all possible coalitions S that are formed without i:
Σ p(S)(o(S U (1) v(S)), p(S) [S)(([ - |S| 1)! SCNVO
d(il)
where p(S) is the likelihood of S being sampled. The Shapley value has been proved be the unique metric that satisfies the following axioms: Linearity, Symmetry, Dummy, and Efficiency [46]. We summarize these axioms in Appendix B.
Shapley Interaction. In the game theory, some players tend to form coalition and always participate in the game together. The players in the coalition might interact cooperate with each other, which

---

brings additional contributions to the game. The Shapley interaction [1] measures this additional contributions brought by the coalition compared with the case when the players work individually. For coalition S, we consider [S] as single hypothetical player, which is the union of the players in S. Then, the reduced game is formed by removing the individual players S from the game and adding [S] to the game. The Shapley value ((S) ) Su {[S])) for player (S] can be computed using Equation over the reduced game, Similarly, we can obtain o( X J {i]), where is the individual player in S. Finally, the Shapley interaction for coalition S is formulated as:
((S) d[S)) \ U ((ST)) - Σ (ill SU (i))
(4)
iES
In this way, J((]) reflects the interactions inside S. The higher value of ([S]) indicates that players in S cooperate closely with each other.
3.2.2 Token-Level Shapley Interaction Modeling
Due to inherent semantic unit mismatch between texts and images, it is ineffeetive to directly compute the alignment between words and pixels (patches), A textual phrase usually refers to specific image region, which is composed of multiple patches and represents a visual instance. Thus, we first introduce token-level Shapley interaction modeling aggregate patches into semantic regions.
Input Representations. Given an image-text pair, the imput image is sliced into patches and flattened. Followed by linear projection layer and position embeddings, we obtain patch token sequence X' = (x/)/2 with an additional [CLS_ token embedding. The input text T is tokenized and embedded into word token sequence X² {xx")<>
[i added with position embeddings. We also prepend a learnable special token [CLS_ T] to the word token sequence. Then, we adopt dual-encoder structure to encode the patch token sequence and word token sequence, separately. On top of the image and text encoders, we obtain the representations of patch token sequence
--= (x ty and word token sequence X²T {x] {xTHE, We take the learned representations of [CLS_I] and [CLS_T] tokens as the global representations for images and texts. And the global similarity of image-text pairs is measured by the cosine similarity between them.
Understanding Semantic Region via Shapley Interaction. Supposing a set of patches represent a complete visual instance in an image, then they tend to have strong Shapley interaction because they work jointly to form a visual instance, which contributes to the better similarity judgment with the text. From the game-theoretic view, take patch tokens and word tokens players X' = X² U X²T and the global similarity between images and texts as the game score 01 (.). To compute U1 (S), we keep tokens in S and mask input tokens in X' S to zeros. Thus, the global similarity only considers the tokens in S, which reflects the contribution of the tokens in S to the global similarity judgment.
Semantic Region Generation. Inspired by YOLOv3 (37, we design lightweight region generation module. takes each patch token representation X as input and generates a bounding box prediction centered on x/, which corresponds to a visual region R2 = region generation module also predicts a confidence score s(R,) for each region. We select the top-M predictions the semantic regions. Then, the Shapley interaction of R, can be defined as:
with K; patch tokens. The
((R.)) <(RRI] \ R2 (R.I))
Σ X,,ER,
o(x{ H
R4 U (x(,))
(5)
According to the Equation
3 we can reformulate Shapley value into the form of expectation:
o (RRIX \ R, U ((RA))
= E{{ E ( - U1 (S)]}
(6)
8 CEARR 18
where e represents the coalition size. o(x/ X R, and the Shapley interaction of R, can be reformulated (we provide the proof in Appendix C):
(x(x)) can be defined in a similar manner,
((R))) = E{_ E (UI() - Σ SCAVR;
vi (x/,)) (K- 1)vi(8))
x ER.
5

---

Taking normalized J'([R:) as the soft supervision label, the token- based semantic aggregation loss defined as cross-entropy loss:
LSA M > (RR))) + 3'((( - 1000-s((20 s(R))))
(8)
which propagates gradients to the region generation module and image encoder to adjust bounding box predictions such that more accurate semantic regions can be captured.
3.2.3 Semantics-Leve Shapley Interaction Modeling
After obtaining the inferred semantic regions, we propose semantics-level Shapley interaction model- ing to explicitly model the fine-grained semantic alignment between regions and phrases. We first define the fine-grained similarity score and then explain semantic alignment based on game theory.
We adopt
ing over learned patch representations in each R; to obtain region representa- tion by E Rd. We employ an off-the-shelf constituency parser to extract phrases from text and obtain phrase representation h. E Rdd by Avg-Pool i ng. Totally, we obtain M regions H = (h M and phrases H" = 1ー And the alignment matrix can defined as:
[a.)] MXN
where 0, = h T hy represents the alignment score between i-th region and j-th phrase. Next, we apply softmax- normalizzaion
each row of A, obtaining Å. For the i-th region, caleulate its maximum alignment score as max, āzy. Then, we use the average maximum alignment score over all regions as the fine-grained image-to-text similarity P1. Similarly, we can obtain the fine-grained text-to-image similarity p2, and the total fine-grained similarity score can be defined:
(2)+22)/2.
Understanding Semantic Alignment via Shapley Interaction. If region and phrase have strong semantic correspondence, then they tend to cooperate with each other and contribute to the fine fin-grrined similarity score. Thus, similarity score p as the game score (2(-). The Shapley interaction of them can be formulated as:
consider H. = 74 UHT as the players and the fine-gratd fine-grained
((H4))) = (1H6,111 Hay U ((G,)1)) ob/// Hly U (h(1) 6(3]M Hyy U (bfT)) (9) (2 U H.y)) (S U (b/}) 02 ( (h, + (S))
El E
(10)
SCNVHy 181=c
where [Hly] represents the single player formed by the coalition of i-th region and j-th phrase. Taking normalized J' ((((H))) as soft labels, the fine-grained semantic alignment loss can be defined as:
M
N
CPSA = MN
(11)
3.3 Uncertainty-Aware Neural Shapley Interaction Learning
According to Equation and Equation 4 computing exact the Shapley value is an NP-hard prob- lem (32]. Previous methods mainly apply sampling-based method [6] to approximate it. While sampling-based approximation is unbiased, an accurate approximation requires thousands of model evaluations. To reduce the computational cost, we propose uncertainty-aware neural Shapley interaction learning (UNSIL) module to cooperate with the sampling-based method, rendering an efficient and effective hybrid strategy.
Specifically, the sampling-based method [6] estimates the expectation terms in Equation 7 and Equation by sampling to compute the Shapley interaction. Inspired by noisy label learning 18] the UNSIL module learns to predict the Shapley interaction and the corresponding uncertainty 1). Intuitively, if the UNSIL module makes prediction with low uncertainty, directly apply its prediction to LTSA and LFSA, avoiding thousands of model evaluations. the uncertainty is high, we then resort the sampling-basee method for
accurate estimation.
During training, the UNSIL module first predicts the target interaction with uncertainty a. Then, we sample value from uniform distribution on (0, 1). If > 0, we directly use its prediction. If

---

Table Results (%) of zero-shot meetee--extte retrieval on Flickr30K and MSCOCO datasets.
Flickr30K
MSCOCO
Image-to-ten
cet--o-inage
Image-to-ten
text-to l0-image
R@ 1
R
R 10
R@1
R
R 10
R@1
R@5
R 10
R@1
R@5
R@
ImageBERT
70.7
90.2
94.0
54.3
79.6
87.5
44.0
71.2
80.4
32.3
59.0
70.2
UNITER
83.6
95.7
97.7
89.2
93.9
88.0
98.7
99.4
90.6
95.2
58.4
81.5
88.1
62.4
72.2
ALIGN
88.6
98.7
99.7
93.8
96.8
58.6
83.0
89.7
45.6
69.8
78.6
FILIP
89.8
99.2
99.8
75.0
93.4
96.3
61.3
84.3
90.4
45.9
70.6
79.3
LOUPE
90.5
99.5
99.8
76.3
93,9
96.7
62.3
85.1
91.2
50.1
75.4
83.3
Table 2: Top-1 accuracy (%) of zero-shot image classification over 11 datasets.
StanfordCar
Flowers102 102
Country211
OxfordPets
Caltech101
CIFARIO
FER2013
ImageNet
Food101
SUN397
Aircrafts
Average
CLIP
96.2
92.9
77.3
67.7
78.7
34.9
57.7
36.1
93.5
92.6
75.3
73.0
LOUPE
95.9
94.3
79.9
69.8
87.4
37.8
53.3
54.9
94.1
93.9
76.1
76.1
é S 0, we then use the sampling- based method to compute the Shapley Interaction and update the UNSIL module based on the campling-basee results. Note that, for the first few iterations, employ the sampling-base method directly, and use its results to train the UNSIL module.
Let J" and 3 denote the results from the sampling-ban baaed method and UNSIL module, respectively. Taking J" as the ground-truth, the UNSII. module is trained by:
CUNSIL B10
(M)E(3, + 820
(12)
where the first term is mean squared eπтоr LMSE weighted by the uncertainty, the second term serves as a regularization term for the prediction uncertainty, and 3 the weight hyper-parameter. The UNSIL module implicitly learns the uncertainty from the regression loss function. We discuss the implementation details of the UNSIL module in Section 4.5 and Appendix D.
4
Experiments
4.1
Pre-training Details
As sufficient data is a prerequisite for vision-language pre-training, we construct dataset with 240M image-text pairs from the Internet. We implement the image encoder by Swin-L (28) and the text encoder by BERT-Small [9] The input images are resized to 224 224 and the input texts are tokenized by WordPiece with maximum length of 60. We pre-train the model for 20 epochs using batch size 512 on 128 NVIDIA V100 GPUs. We utilize Adam [29] optimizer with a learning rate of 2 x 10-4 and weight decay of 0.01. More pre-training and evaluation details provided in Appendix D, E. We also analyze the image encoder and training efficiency in Appendix G.
4.2 Zero-Shot Image-Text Retrieval
We compare LOUPE on the widely used MSCOCO 27 and Flickr30K 33 datasets. First, the results in T that LOUPE achieves new of-te-ofthhe-rr zero-shot performance on most metrics of the two datasets, demonstrating the stronger generalizability of our pre-training framework. Second, while previous works mainly pre-train on larger datasets (CLIP 400M, ALIGN 1800M, FILIP 340M), LOUPE still achieves superior performance using less training data (240M). Third, compared with FILIP which directly computes token-wise similarity, our model captures semantic alignment between visual regions and textual phrases, which is more semantically meaningful. For text-to-image retrieval on MSCOCO, LOUPE significantly outperforms FILIP by 4.2% on recall@) 1.
4.3 Zero-Shot Image Classification
In this section, we evaluate LOUPE on the zero-shot image classification task. compare LOUPE with CLIP 11 downstream classification datasets, following the same evaluation setting as
7

---

Table 3: Without fine-tuning, zero-shot transfer performance on object detection and visual grounding.
COCO
PASCAL VOC
ReiCOCO
RelCOCO+
m.
mAP@0.5
mAP@0.3
m.
val
lestA
testB
val
testA
testB
CL.PP Wise
8.5
18.2
6.7
6.2
7.0
CLIP K-Meuns
6.4
1.9
11.7
4.8
2.1
2.3
1.7
1.7
2.0
2.8
CLIP Grad-CAM
7.1
3.2
19.1
8.2
5.5
5.2
4.8
4.4
5.6
4,9
AdapiCL.IP LOUPE
14.9
6.6
28.7
16.7
18.4
18.0
17.5
18.9
19.6
25.3
12.1
26.8
23.3
photo of Porson
Church
photo Dog
Text
photo of Church
Prompt
Raaaos-Region Semantic Alignment
Image
Semantic Region
Doe
Figure 2: An example of LOUPE zero-shot transferring to object detection using prompt templates.
Table 2 summarizes the results, As shown in Table 2 our LOUPE outperforms CLIP with average improvement of 3.1%. Notably, on ImageNet, the largest dataset among 11 datasets, our LOUPE surpasses CLIP by 0.8%. Also, we observe that LOUPE achieves substantial perfor- mance gains on several fine-grained image classification datasets (i.e., Flowers102 and Airerafts). It demonstrates the superiority of our LOUPE on mra-grained semantics understanding.
CLIP 35
We also evaluate the linear probing performance of our LOUPE on image classification. The detailed results can be found in Appendix I.
4.4 Zero-Shot Transfer to Object Detection and Visual Grounding
To answer whether model has learned fine-grained semanties, further evaluate LOUPE on object detection [39] and visual grounding (51), which require more fine-graine tne-grained semantic understanding ability identify specific visual regions images according to the object labels or referring expressions. Visual grounding can be seen as generalized object detection, where the pre-defined class labels are replaced by language referring expression sentences. As LOUPE can generate a set of semantic regions that are aligned with textual phrases, it can be easily applied to object detection and visual grounding without structure modification. For visual grounding, we take referring expressions as input text. For object detection, illustrated in Figure 2 expand detection labels to input text. Then, we encode input text by the learned text encoder, and these tasks can be completed by measuring the similarity between candidate region representations and text representations.
we use prompt to
For comparison, we zero-shot transfer CLIP (ViT-L/14) to object detection and visual grounding by applying several non-parametric approaches on the spatial feature maps of CLIP. We also compare with AdaptCLIP [21]. which is a concurrently unpublished method that leverages classic super- pixel (SLIC I and bounding box proposal (selective search 44 methods to zero-shot transfer CLIP to phrase localization. We use its public official implementations to get the experiment results. For object detection, we evaluate their mean Average Precision (mAP) at IoU thresholds of {0.3, 0.5) on COCO (27 (65 classes) and PASCAL VOC [1I (20 classes). For visual grounding, we evaluate their top-1 accuracy at loU thresholds of 0.5 on RefCOCO [51 and RefCOCO+ (1]. The experiment details of CLIP variants and LOUPE are provided in Appendix E.
Table summarizes the results. 1) Overall, LOUPE outperforms all CLIP variants by a large margin. The significantly higher performance illustrates the stronger zero-shot transfer ability of our fine-grained semantically aligned pre-training paradigm. 2) Second, all CLIP variants rely on pre-processing steps on CLIP's feature map (e.g., AdaptCLIP first uses SLIC to group pixels and then uses selective search generate a large number proposals), which is consuminn, In contrast, our method directly predicts the semantic regions based on the patch token representations. 3) Third, consistently competitive performance across four benchmarks validates that LOUPE can learn fine-grained semantics from raw text supervision. LOUPE demonstrates promising alternative, that is, learning fine-grained semantics from large-scale raw image-text pairs, which are easily available and contain a broader set of visual concepts.

---

Table Ablation study of each component across three tasks.
MSCOCO
COCO
ReiCOCO
Truining Time
12T
T21
mAP@0.3
mAP@0.5
lestA
testB
(sec/iter)
Backbone
31.0
24.8
3.8
1.0
1.3
0.9
0.8
1.17
2 Backbone CTSA
32.4
26.2
7.6
3.3
1.8
2.0
2.6
8.38
3
Backbone LTSA + LESA
33.5
28.3
9.4
5.9
4.1
4.6
4.3
9,90
Backbone LISA + LESA + UNSIL
33.3
28.1
9.0
5.6
4.5
4.9
4.4
1.93
6:122
0:99
6.120
0.22
9.12
0000
0.05
6015
6.3
200
1900
1585
2000
2006
1000
6:)
Cohr10
MIP
Transbyrer
9.60
GowrD
MLP
Tondoomer
Sampling Number
(a) Instability
(b) Uncertainty
(c) Error
Figure 3: (a) Instability of the Shapley interaction approximation with respect different sampling numbers. (b, c) Uncertainty and error of the UNSIL module with different structures.
As time-consuming human annotations are unscalable for massive object classes in the real world, some recent works 36 target at training object detectors with annotations on base object classes generalize to the remaining object classes of the same dataset. The latest works [3] 52 leverage the generalizability of vision-language pre-training models further improve the zero-shot performance on novel classes. However, these zero-shot approaches still require bounding box annotations on base classes for task-specific supervised learning. In contrast, our LOUPE is trained on large-scale raw image-text pairs, which are already accessible on the Internet and contain more diverse semantics.
4.5 Ablation Study
Effectiveness of Individual Components. In this section, we investigate the effectiveness of each component in Table 4 Given the costly training time, all ablation studies are based on a relatively small dataset (Conceptual Captions 3M (41). We start with the backbone model that consists of a dual-encoder trained by cross-modal contrastive loss. We then gradually add token- level Shapley interaction modeling supervision CTSA (Row 2), semantics-level Shapley interaction modeling supervision LFSA (Row 3), and UNSIL module (Row 4). For Row 2 and Row 3, the Shapley interaction is only computed by the sampling-basee based method. The results in Table show that both CISA and LFSA bring significant improvement for all tasks. We observe that LISA boosts 3.8% improvement on object detection. And the improved fine-grained visual semantic understanding further facilitates the cross cross-modal retrieval performance (+1.4%). The semantics Shapley interaction modeling further improves the performance on all tasks by modeling the semantic alignment between visual regions and textual phrases. Comparing Row 3 and Row 4, we observe that the UNSIL module maintains the estimation accuracy while avoiding intensive computations. The averaged training time is reduced from 9.90 seconds per iteration to 1.93 seconds per iteration.
Accuracy of the Shapley Interaction Learning. Since use the sampling-bai based method 6 to compute the Shapley Interaction and train the UNSIL module, we conduct a study to evaluate the accuracy of the sampling-bas based method and the eтror of the UNSIL module. As |54 we compute the interaction multiple times and measure the instability of them. A lower instability means that we obtain similar interactions from different sampling processes. It indicates a high accuracy. Specifically, the instability is defined as Eu. where J., denotes the interaction computed in the w-th E..]3...
time. We average the instability values over Shapley interaction of 100 image-text pairs. We report the average instability values with respect to different sampling numbers. shown in Figure 3 (a), the instability decreases along with the increase of the sampling number. When the sampling number is larger than 500, the approximated Shapley interaction stable enough with instability less than 0.06. Further, attempt different models (i.c., ID, 3-Layer MLP Attention, 3-Layer Transformer) to implement the UNSIL module (please see Appendix D for more details). We test them 1000 samples and report their mean uncertainty and relative error in Figure (b) and (c). We observe that MLP + Attention good enough to predict the interaction with lower complexity. Thus, implement the UNSIL module by MLP Attention.

---

This paper introduces a novel vision-language pre-training framework, LOUPE, which models the fine-grained semantic alignment between visual regions and textual phrases by same-theoretic interactions. To efficiently compute the interactions, we further propose an uncertainty- Shapley interaction learning module. Comprehensive experiments show that LOUPE achieves new state-of-the-ae on image-text retrieval datasets and can transfer to object detection and visual promising direction of learning with umbrelia
relphart 5,50
person 0.15
zebra
JO prote
Whlll sog
(a) Object Detection
(b) Visual Grounding
Figure 4: Qualitative examples of object detection on COCO and visual grounding on RefCOCO+
Region
Rocion
Ropion
Haggan
Flegan
Recton
Région
6:37
0.40
0.7c
A cat tred to doer snen with blin bag over with Sabas the Wonevio
Cal Bag
A Aggiii playing with a dog on dhali with 8 ysllow bag boside Fiem.
A diag on it lnseet atteched to
wopden bendh
Algement
Figure 5: Visualization of learned fine- -hee-graied semantic alignment and corresponding Shapley interaction values. The values in the red boxes represent Shapley interaction of regions.
4.6 Qualitative Analysis
Qualitative Examples. As shown in Figure 4 LOUPE successfully captures the regions that correspond to the detected objects, and grounds the referring expressions onto the referred regions.
Visualization of Learned Fine-Grained Semantic Alignment. In Figure 5 we visualize some key semantic regions and corresponding alignment matrices inferred by LOUPE. We present the regions with top-3 confidence (Region 1 -3) and two randomly sampled regions (white boxes). The red boxes the bottom of bounding boxes indicate their normalized token-level Shapley interaction values. Comparing their Shapley interaction values, we observe that the token-level Shapley interaction successfully distinguishes semantic regions from randomly sampled regions. The semantically meaningful regions tend to have stronger interaction. It indicates that token-level Shapley interaction can provide correct supervision for semantic region generation. Further, we show the alignment matrices inferred by semantics-leww Shapley interaction and LOUPE, respectively. As shown in the and aligns it with the "a leash" phrase. Note that existing object detection datasets do not contain the "leash" category.
right case of Figure (5] LOUPE successfully recognizes the leash region
5 Conclusion
neural
grounding in a zero-shot manner. This work demonstrates fine-grained semantics from large-scale raw image-text data.
Limitations. 1) The phrases extracted by off-the-shelf constituency parsers, whose predictions might not be completely accurate. 2) The web data might inevitably contain mismatched image-text pairs, leading noisy supervision.
Social Impacts. Our model is trained on noisy data from the Internet, which may contain unsuitable images, violent text, or private information. Thus, additional analysis of the data is necessary. Further, the use of our model for privacy surveillance or other nefarious purposes should prohibited.
Acknowledgment. This work has been supported in part by the National Key Research and Devel- opment Program of China (2018AAA0101900) Zhejiang NSF (LR21F020004), Key Research and Development Program of Zhejiang Province, China (No. 2021C01013), Chinese Knowledge Center of Engineering Science and Technology (CKCEST). We thank all the reviewers for their valuable comments.
10

---

References
[1] Radhakrishna Achanta, Appu Shaji, Kevin Smith, Aurelien Lucchi, Pascal Fua, and Sabine Süsstrunk. Slic superpixels compared to state-of-the-art superpixel methods. IEEE transactions pattern analysis and machine intelligence, 34(11):274422282, 2012.
[2] Peter Anderson, Xiaodong He. Chris Buchler. Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In Proceedings of the IEFE conference on computer vision and potter recognition, pages 6077-6086, 2018. [3] Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision. pages 2433, 2015.
[4] Ankan Bansal, Karan Sikka, Gaurav Sharma, Rama Chellappa, and Ajay Divakaran. Zero-shot object detection. In Proceedings of the European Conference on Computer Vision (ECCV), pages 384-400, 2018. [5] Tom Brown, Benjamin Mann, Nick Ryder. Melanie Subbiah. Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot leameer.
Advances in neural information processing systems, 33:1877-1901, 2020.
[6] Javier Castro, Daniel Gómez, and Juan Tejada. Polynomial calculation of the shapley value based on sampling. Computers & Operations Research, 36(5): 126-).17 2009.
[7] Yen-Chun Chen. Linjie Li. Licheng Yu, Ahmed EI Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing
Liu. Uniter: Universal image-text representation learning. In European conference on computer vision,
pages 104-120. Springer, 2020.
[8] Anupam Datta, Shayak Sen, and Yair Zick. Algorithmic transparency via quantitative input influence: Theory and experiments with learning systems. In 2016 IEEE symposium on security and privacy (SP). pages 598-617. IEEE, 2016.
[9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arXiv: 1810.04800, 2018.
[10] Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold. Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. / //929, 2020. [11] Mark Everingham, Andrew Zisserman, Christopher KI Williams, Lue Van Gool, Moray Allam, Christo- pher M Bishop. Olivier Chapelle, Navneet Dalal, Thomas Deselaers, Gyuri Dorkó, et al. The pascal visual object classes challenge 2007 (voc2007) results. 2008
[12] Michel Grabisch and Mare Roubens. An axiomatic approach to the concept of interaction among players in cooperative games. International Journal of game theory, 28(4):547-565, 1999.
[13] Xiuye Gu. Tsung-Y Yi Lin, Weicheng Kuo, and Yin Cui. Open-vocabulle object detection via vision and
language knowledge distillation. arXiv preprint arXiv:2104.132: 13921, 2021.
[14] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729-9738. 2020.
[15] Zhicheng Huang. Zhaoyang Zeng. Yupan Huang. Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision-language representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12976-12985 1295,, 2021.
[16] Zhicheng Huang. Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image
pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004. 00849, 2020.
[17] Chao Jia, Yinfei Yang. Ye Xia. Yi-Ting Chen, Zarana Parekh, Hicu Pham. Quoe Le.
Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pages 4904-4916. PMI.R, 2021.
[18] Alex Kendall and Yarin Gal. What uncertainties do we need in bayesian deep learning for computer vision? Advancey in neural information processing systems, 30. 2017.
[19] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and- lan-uagge transformer without convolution or region supervision. In International Conference on Machine Learning, pages 5583-5594, PMLR, 2021. Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems. 34, 2021.
[20]
[21] Jiahao Li, Greg Shakhnarovich, and Raymond A Yeh. Adapting clip for phrase localization without further
training. arXiv preprint arXiv:2204.0 03644, 2022.
[22] Juncheng Li, Xin Wang, Siliang Tang, Haizhou Shi, Fei Wu, Yueting Zhuang, and William Yang Wang. Unsupervised reinforcement learning of transferable meta-skills for embodied navigation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 12123-12132. 2020.
[23] Liunian Harold Li. Pengchuan Zhang. Haotian Zhang, Jianwei Yang. Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan. Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-iman image pre-training. arXiv preprint (tt:2212.00882 2021.
[24] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang. Unimo: Towards unified-modal understanding and generation via cross- contrastive learning. arXiv preprint
arXiv:2012. 15409, 2020.
[25] Xiujun Li, Xi Yin. Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong
Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for rision-language tasks. In
11

---

European Conference on Compuder Vision, pages 121-137. Springer, 2020.
[26] Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language- image pre-training paradigm.
arXiv preprint arXiv:2 10.05208, 2021.
[27] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755. Springer, 2014.
[28] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang. Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF
International Conference on Computer Vision (ICCV), 2021.
[29] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
1r11v:1711.05101 2017.
[30] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostie visiolinguistic representations for ssion-anddlanguage tasks. Advances in neural information processing systems, 32, 2019.
[31] Scott M Lundberg and Su-In Lee. A unified approach to interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[32] Yasuko Matsui and Tomomi Matsui. Np-completeness for calculating power indices of weighted majority games. Theoretical Computer Science, (6233333 310, 2001.
[33] Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan C Caicedo, Julia Hockenmaicr, and Svetlana Lazebnik. Flickr30k entities: Collecting eegion-to-prrase correspondences for richer image-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pages 2641-2644 2015.
[34] Di Qi. Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. Imagebert: Cross-modst pre-training with laage-scale weak supervisedd image- data, arXiv preprint orXiv:2001.0P 0,09660 2020.
[35] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 3748-8763. PMLR.
2021.
[36] Shafin Rahman, Salman Khan, and Nick Barnes, Improved object detection. In Proceedings of the AAAI Conference on Artifiiaal Intelligence, volume 34, pages
alignment for zero-shot
11932-11939, 2020.
[37] J. Redmon and A. Farhadi. Yolov3: An incremental improvement. arXiv e-prints, 2018.
[38] Jic Ren, Die Zhang, Yisen Wang, Lu Chen, Zhanpeng Zhou, Yiting Chen, Xu Cheng, Xin Wang. Meng Zhou, Jie Shi. et al. A unified game-the interpretation of adversarial robustness. arXiv preprint
arXiv:211..03588
2021.
[39] Shaoqing Ren, Kaiming Hc, Ross Girshick, and Jian Sun. Faster r-enn: Towards real-time object detection
with region proposal networks. Advances in neural information processing systems, 28, 2015.
[40] Lloyd S Shapley. A value for n-person games, contributions the theory of games, 2, 307-317, 1953.
[41] P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt-
text datasct for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume 1: Long Papers), 2018.
[42] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vl-bert: Pre-training of generic visual- -oual-linguistic representations. arXiv preprint 1rrXivI908..00 085300 2019.
[43] Hao Tan and Mohit Bansal. Lxmert: learning cross-modality encoder representations from transformers. arXiv preprint arXiv: 07490, 2019.
[44] Jasper RR Uijlings, Koen EA Van De Sande, Theo Gevers, and Arnold WM Smeulders. Selective search
for object recognition. International journal of computer vision. 104(2):54-171, 2013.
[45] Ashish Vaswani. Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems,
30, 2017.
[46] Robert J Weber. Probabilistic values for games. The Shapley Value. Essays in Honor of Lloyd S. Shapley, pages 101-119, 1988.
[47] Longhui Wei, Lingxi Xie, Wengang Zhou, Hougiang Li, and Qi Tian. Myp: Multimodality-uuddee visual pre-training. arXiv preprint arXiv:2203.051 220..077... 2022.
[48] Kelvin Xu, Jimmy Ba. Ryan Kiros, Kyunghyun Cho, Aaron Courville, Ruslan Salakhudinov, Rich Zemel, and Yoshua Bengio. Show, attend and tell: Neural image caption generation with visual attention. In International conference on machine learning, pages 2048-2057. PMLR, 2015.
[49] Lewei Yao, Runhui Huang. Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre- pre-training. arXiv preprint
mrXiv:2111.0778. 2021.
[50] Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-vil: Knowledge enhanced vision-language representations through scene graph. arXiv preprint arXiv:2006. 16934, 2020. [51] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European Conference on Computer Vision, pages 69-85. Springer. 2016.
[52] Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu, and Shih-Fu Chang. Open-vocabulary object detection using captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pages 14393-14402, 2021.
12

---

[53]
Yan Zeng. Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. arXiv preprint arXiv:2111.08. 2021.
[54]
Hao Zhang. Yichen Xic, Longjie Zheng, Die Zhang, and Quanshi Zhang. Interpreting multivariate shapley interactions in dnns. arXiv preprint axXiv2001..00445 2020.
[55] Pengchuan Zhang. Xiujun Li, Xiaowei Hu, Jianwei Yang. Lei Zhang. Lijuan Wang. Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 5579-5588, 2021.
[56] Yiwu Zhong, Jianwei Yang, Pengchuan Zhang, Chunyuan Li, Noel Codella, Liunian Harold Li, Luowei Zhou, Xiyang Dai, Lu Yuan, Yin Li, et al. Regionclip: Region-basee language-image pretraining. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16793-
16803, 2022.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Please see Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please see
Section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes] Please see Appendix B,
C.
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] The pre-trained LOUPE model is coming soon.
(b) Did you specify all the training details (c.g., data splits, hyperparameters, how they were chosen)? [Yes] Please see Section 4.1 and Appendix D, E.
(c) Did you report error bars (e.g.. with respect to the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] Please see Section 4. and Appendix D, J.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [Yes]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [No] We do not use crowdsourcing conducted research with human subjects.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [No] We do not use crowdsoureing or conducted research with human subjects.
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [No] We do not use crowdsourcing or conducted research with human subjects.
14


Reviews:
[{"review_id": "ygKFprg6Iy5", "reviewer": "Reviewer_7KYj", "summary": "This paper focuses on learning fine-grained vision-language alignment under the widely used contrastive-based pre-training paradigm. The authors first point out that previous pre-training frameworks, *e.g.,* CLIP, only models global image-text alignment while neglecting the fine-grained semantic features. Alignments though can be achieved by interacting between modalities with the help of certain mechanisms like cross-attention, it lacks explicit and strong supervision to encourage meaningful patch-phrase correspondence, which proved to be quite useful for reasoning downstream tasks. The authors address the limitation by explicitly matching phrases in the text with regions of patches in the image. Different from previous approaches, the authors design light modules to achieve patch-to-patch and region-to-phrase alignment by introducing supervisions derived based on the Shapley value in game theory. Practically, Shapley interaction values are used as the criteria to estimate whether two patches belong to the same region and whether an image region corresponds to a certain phrase. In addition, to address the training efficiency problem when computing Shapley values, an additional estimator is adopted to predict the true values. Extensive experiments show the zero-shot ability of the proposed model on retrieval tasks, detection tasks, and visual grounding tasks. Ablative studies on the new losses and Shapley value approximation module are conducted. Limitations and social impacts are also discussed.\n\nOverall, this paper shares an interesting idea by incorporating game theory mechanisms in the process of text/image alignments. This kind of supervision is free of pre-trained detection/segmentation models and does not require sophisticated text/image labels. The proposed model shows significant improvements on the zero-shot transfer ability and outperforms baseline approaches by a large margin.", "questions": "I have stated my major concerns in the **Weaknesses** section. The following are some detailed questions and suggestions.\n\n+ How is the score function in the game defined? Does it need to be carefully designed?\n+ The alignment between image regions and text phrases implicitly involves the concept of object. This may lead to similar limitations to the previous works that relied on pre-trained object detectors, and hard to generalize to object-free inputs. Also, according to line185, all regions are constrained within a scale of K patches. I wonder how does the model deal with objects (concepts) of different scale, *e.g.,* apple and sky?\n+ Line 230. How long do we need to train the UNSIL module?\n+ I am confused about Eq.(12). The authors claim the loss function is derived from the regression loss function. Could the authors provide a detailed derivation process? From my perspective the second term $\\beta_2 \\ \\sigma$ should be $\\beta_2 \\ {\\rm log}(\\sigma)$.\n+ I am also concerned about the training time shown in table 3. It seems like the additional cost is still huge even with the approximation module. I wonder if the authors can provide a training cost-performance trade-off compared with other works? Some qualitative analysis would also do the trick.\n+ If possible, I would like to see a comparison between the proposed model and some missing related works, like RegionCLIP and X-VLM that were mentioned above. These approaches share similar insight and can also perform zero-shot transfer to several downstream tasks.", "limitations": "The authors have adequately addressed the limitations and social impacts in the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ Strengths\n  + This paper is well organized and easy to follow. The authors first give a brief introduction on the concept of Shapley values and Shapley interaction values and elaborate on how it works under the framework of multi-modal pre-training. The derivation seems to be sound and easy to understand. The main results and ablative studies help to better understand the proposed model.\n  + The idea in this paper is quite interesting and seems novel to me. The idea of aligning image regions with text phrases has been explored in previous works, while most of them rely on additional label information and fail to provide strong supervision. Introducing Shapley value in the aligning process sounds reasonable and practically shows promising results.\n \n+ Weaknesses\n  + Some missing related works. Similar to this paper, there are also works that attempt to address the problem of missing fine-grained information. For example, RegionCLIP[1] also finds regions in the image and aligns them with text phrases. Off-the-shelf language parsers are similarly considered. Also, RegionCLIP is able to perform zero-shot object detection tasks. Another work, X-VLM[2], shares a similar insight. I believe these works are more related to the paper, and a detailed discussion on the connections and differences with these works is suggested.\n  + Training efficiency. I understand that the authors use a sub-network to predict the real Shapley value to save excessive computation costs. Nevertheless, the training time in table 3 (comparing line 1 and line 4) shows that the proposed approach still involves 60~70 training time. This can be a great burden, especially for large-scale pre-training.\n  + Please see the **Questions** section for some detailed concerns.\n\n[1] Zhong, Yiwu, et al. \"Regionclip: Region-based language-image pretraining.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n[2] Zeng, Yan, Xinsong Zhang, and Hang Li. \"Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts.\" arXiv preprint arXiv:2111.08276 (2021).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "xzaspfuARWx", "reviewer": "Reviewer_gkzN", "summary": "This paper proposed a fine-grained semantically aligned vision-language pre-training framework. Both visual token-level semantics alignment and phrase-region level semantics alignment are studied. To be more specific, the paper measures the shapley interaction of visual regions to text as a supervision signal for region generation module and visual backbone. Then the alignment between regions and phrases is also supervised by Shapley interaction between them. Further, an uncertainty-aware learning module is introduced to predict Shapley interaction to save the time cost of sampling-based estimation. ", "questions": "Please refer to weaknesses.", "limitations": "It's not guaranteed that the dataset they use doesn't contain unsuitable images, text, or personal information. ", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The methodology is interesting and novel. It's essential to exploit the local correlation between image and text in contrastive pre-training. The idea of introducing Shapley interaction as supervision seems to fit the problem well.\n2. LOUPE can outperform recent works trained with a similar scale of data in retrieval tasks. And the ablation is comprehensive.\n3. The region generation model can also be used as a zero-shot object detector, which is quite impressive.\n\nWeaknesses:\n1. The evaluations on zero-shot image classification and linear probing are missing. Those are important to prove whether the learned visual backbone is strong and robust.  In CLIP, they reported the above two evaluations in 20+ datasets to prove the transferability of their visual backbone. \n2. The training time needed is increased by 65% (1.17->1.93). \n3. The dataset used seems to be a private dataset and it's hard to convince people that the comparison is fair enough. Although the scale of the proposed dataset is smaller, the quality is not compared and it matters a lot in CLIP-like pretraining. Also, it's not indicated whether the dataset will be released or not. If not, it's hard for people to reproduce this work. \n\n================================\nAfter rebuttal, weakness 1 and 2 have been alleviated. So I change the score.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "VZqC7EKJ4wI", "reviewer": "Reviewer_H9ch", "summary": "This paper propose a fine-grained semantically aligned vision-language pre-training framework（LOUPE） from game-theoretic interactions. Experiments on image-text retrieval, object detection and visual grounding tasks demonstrate the effectiveness of LOUPE. \n", "questions": "1. The experiments are not sufficient. The model structures（both image encoder and text encoder) of different methods in Tabel 1 and Table 2 are inconsistent and the comparison in unfair.\n\n2. Can plain Dual-Encoder be used for zero-shot classification? How is the result on ImageNet dataset?\n\n3. Why are the loss weights of $L_{CMC}$, $L_{TSA}$ and $L_{FSA}$ set to 1:1:1?\n\n4. Are public datasets, such as CC12M, included in the 240M dataset? In addition to the description part in Supplementary Material E, it is hoped that a more detailed introduction to the 240M dataset can be provided.", "limitations": "The authors have adequately addressed the limitations and potential negative societal impact of their work", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. The motivation is clear. LOUPE propose Phrase-Region Semantic Alignment to achieve fine-grained vision-language pre-training.\n\n2. The hybrid Shapley interaction learning strategy is interesting for me.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "IJKVhnV6w-a", "reviewer": "Reviewer_yCwd", "summary": "The paper addresses the problem of vision-language pretraining, whose goal is to learn a strong backbone network transferable into downstream tasks such as object detection or visual grounding.\nThus, the paper proposes a framework for aligning between visual and textual tokens during training motivated by Shapley value from game theory.\nSpecifically, the model computes the Shapley values to determine the interaction between related image patches, which form objects or related image regions and noun phrases corresponding to object types in images.\nThe Shapley values for patches in image regions and pairs of image regions and noun phrases are used as soft pseudo-labels to train the alignment between textual and visual modalities.\nThe paper also introduces an approximation mechanism to reduce the complexity of Shapley value computation via sampling.\nThe paper evaluates the performances of Image-Text Retrieval, Object Detection, and Visual Grounding on MSCOCO and Flickr30k datasets.\n\n------- After rebuttal -----\n\nThe rebuttal sufficiently addresses my concern as well as confirming my understanding.\nThus, I keep my rating and recommend the paper for acceptance.", "questions": "+ Please refer to the weakness section.\n", "limitations": "+ I believe the work doesn't seem to have any potential negative societal impact.\n+ It would be interesting to investigate the bias from web data in future directions. Would the model propagate these biases from captioned images toward downstream tasks?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "#### Strength:\n+ The formulation of semantic generation as well as semantic alignment under Shapley interaction is interesting and novel. Moreover, viewing weakly supervised visual-textual alignment in terms of game theory is a promising direction to explore.\n+ Learning to predict Shapley more, as well as its uncertainty, seems to significantly reduce the training complexity.\n+ The paper compares with appropriate baselines to demonstrate its effectiveness.\n\n#### Weakness:\n+ The technical section could be challenging to follow, especially for computer vision audiences without prior knowledge of Shapley value. The reviewer believes it would be very helpful if the paper could include a table to highlight key differences of the proposed work with various methods used for vision-language pretraining. If the reviewer understands correctly, the main distinction compared to other works would be the proposal of using Shapley interaction as soft pseudo labels for fine-grained image regions instead of using contrastive learning between image-caption pairs. However, this is not clear and emphasized enough in the paper.  \n+ The reviewer is unclear how Shapley value could be used as soft pseudo labels. For example, in line 180, the paper mentions that the game score $v_1$, which is used to compute the Shapley interaction, is the similarity between image and text. However, it appears that the paper does not mention any training phase for learning image-textual similarity. Then how would this similarity be computed correctly? Are there any pretraining phases or pretrained models employed for similarity computation? \n+ The paper lacks convincing motivation on why using Shapley value instead of token-wise alignment (FILIP [42]) is beneficial. Although in lines 97-99, the paper mentions that FILIP has quadratic complexity, the proposed method also suffers from combinatorial complexity, which could be worse. \n+ Although the idea of directly learning to approximate the Shapley value is interesting, it is unclear from the paper how the module can learn to approximate Shapley interaction and estimate its uncertainty. Specifically, what is the input of this module? Is it the set of all visual+textual tokens? Is the uncertainty estimation reliable as neural networks are notorious for being very confident in their predictions?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept