PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining
Authors: Yuting Gao
Jinfeng Liu
Zihan Xu
Jun Zhang
Ke Li
Rongrong Ji
Chunhua Shen
Venue: NeurIPS 2022 Accept (2022)
Keywords: Vision-language Pre-training

Abstract: Large-scale vision-language pre-training has achieved promising results on downstream tasks. Existing methods highly rely on the assumption that the image-text pairs crawled from the Internet are in perfect one-to-one correspondence. However, in real scenarios, this assumption can be difficult to hold: the text description, obtained by crawling the affiliated metadata of the image, often suffers from the semantic mismatch and the mutual compatibility. To address these issues, we introduce PyramidCLIP, which constructs an input pyramid with different semantic levels for each modality, and aligns visual elements and linguistic elements in the form of hierarchy via peer-level semantics alignment and cross-level relation alignment. Furthermore, we soften the loss of negative samples (unpaired samples) so as to weaken the strict constraint during the pre-training stage, thus mitigating the risk of forcing the model to distinguish compatible negative pairs. Experiments on five downstream tasks demonstrate the effectiveness of the proposed PyramidCLIP. In particular, with the same amount of 15 million pre-training image-text pairs, PyramidCLIP exceeds CLIP on ImageNet zero-shot classification top-1 accuracy by 10.6%/13.2%/10.0% with ResNet50/ViT-B32/ViT-B16 based image encoder respectively. When scaling to larger datasets, PyramidCLIP achieves the state-of-the-art results on several downstream tasks. In particular, the results of PyramidCLIP-ResNet50 trained on 143M image-text pairs surpass that of CLIP using 400M data on ImageNet zero-shot classification task, significantly improving the data efficiency of CLIP.

PyramidCLIP: Hierarchical Feature Alignment for Vision-language Model Pretraining
Yuting Gao
Jinfeng Liu 122,+
Zihan Xul
Jun Zhang'
Ke Li
Chunhua Shen
Teencent Youtu Lab
Shanghai Jiaotong University
3 Zhejiang University
(yutinggao, i ianxxu, bobby jzang, tristanli)@teeccnt.com
Abstract
Large-scale vision-language pre-training has achieved promising results on down- stream tasks. Existing methods highly rely the assumption that the image-text pairs crawled from the Internet are in perfect one-to-one correspondence. However, real scenarios, this assumption can be difficult to hold: the text description, obtained by crawling the affiliated metadata of the image, often suffers from the semantic mismatch and the mutual compatibility. To address these issues, we introduce PyramidCLIP. which constructs an input pyramid with different semantic levels for each modality, and aligns visual elements and linguistic elements the form of hierarchy via peer-level semantics aligument and cross-level relation alignment. Furthermore, we soften the loss negative samples (unpaired samples) as weaken the strict constraint during the pre-training stage, thus mitigating the risk forcing model to distinguish compatible negative pairs. Experiments five downstream tasks demonstrate the effeetiveness of the proposed Pyramid- CLIP. In particular, with the same amount of 15 million pre- pre-training image-text pairs, PyramidCLIP exceeds CLIP ImageNet zero-shot classification top-1 accuracy by 0ete/....%/1..... with ResNet50/ViT SE1/50//VT-B32//V1 ITTTBIG 16 based image encoder respectively. When scaling to larger datasets, PyramidCLIP achieves the state-of-the-ar results on several downstream tasks. In particular, the results SiammidCIIP-ReeNCC
trained 143M image-ten text pairs surpass that of CLIP using 400M data on ImageNet zero-shot classification task, significantly improving the data efficiency of CLIP.
Introduction
Recently, vision-language pre- vee-training (VLP) has achieved great success, which aims to improve the accuracy of downstream vision-language tasks by pre-training a model on large-scale image-text pairs harvested from the web without any manual annotation. The mainstream VLP methods can be categorized into two paradigms, single-stream (1; 3; 4; 5) and dual-stream (6; 7; 8; 9; 10). Compared to the single-stream counterpart, the dual-stream paradigm decouples the image encoder and text encoder and extracts features for images and texts respectively. Due its simplicity and flexibility for downstream applications, the dual-stream paradigm dominates. The representative dual-stream model CLIP (6) performs contrastive text pairs collected from the Internet, which achieves astounding results. Later, DeCLIP (10) and FILIP (8) improve CLIP by introducing more supervisions, and bringing in fine-grained modal interaction.
pre-training on 400M image-
Although existing CLIP-alike methods have achieved promising results on downstream tasks, they strongly rely on the assumption that the image-text pairs are of high quality, iee. the pairs are in
'The first three authors contributed equally. This work done when Liu intern Tencent.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

perfect one-to-om one correspondence and have no correlation with other umpaired samples. However, this ideal assumption hard satisty in practice as shown in Figure Firstly, semantic mismatch
between the visual modality and linguis- tie modality exists, e.g., (a) Caption Re- dundancy: the affiliated text description is redundant and contains irrelevant informa- tion; (b) Image Redundancy: the Region- of-Interest (ROI) corresponding to the text is only a sub-region of the image; and (c) Cast Deficiency: text misses the descrip- tions of main objects in the image, while visual modelling needs to reason about the relationship among salient instances. Sec- ondly, captions might be compatible to some extent among pairs, as illustrated in Figure ((d). Existing methods directly treat other pairs as negative samples regardless the potential correlation, which may con- fuse the model.
The sandy keach nor
the apertmest.
chidd roaing book.
Pyeryin of family lovex there.
Captia Redindaary siater
(b) Imagy Bedhnndury
Winter Wodsing at
Boodsics
Glao, foor
Mangrooe andyy boadh.
Mutuall Computibilty
Figure 1: Problems in the web-crawled image-text pairs. (a)()(c) suffer the semantic mismatch between visual modal- ity and linguistic modality, while (d) shows example of the mutual compatibility with (a). Note that, in (a) the red caption is redundant; (b) the image outside the red bounding box the redundant; in (c) the descriptions for the casts in the red boxes are missing: and in (d) the red caption is compatible with the image of (a).
In order to tackle the issues mentioned above, we propose PyramidCLIP in this paper, which attempts to align image-text pairs more precisely in the form of hierarchy. PyramidCL.IP constructs two input pyramids with different se- mantic levels at both sides of the dual-stream network, i.e., the global image, local image region, and ROI features of the salient objects in the image for visual modelling; text summarization, the original caption and categories with attribute descriptions of salient objects for linguistic modelling. Then we contrast visual elements and linguistic elements via peer-level semantics aligmment and cross-level relation alignment, tackling the issues of mentioned in (a), (b) and (c). Specifically, for peer-level semantics alignment, since the global image and text summarization both contain global semantic information, and the local region and original caption both contain more fine- aeee-graind semantic infor- mation, they are treated as two pairs of positive samples. For cross-level relation alignment, to avoid the modelling of object relationship being overwhelmed by the semantics modelling, we explicitly align the fine-grained object relation with cross-layer elements in another modality. Moreover, for the issue of the mutual compatibility, we soften the loss term of the negative unpaired samples during the contrast process to ease the strict constraint, alleviating the negative effect of unpaired similarities.
Extensive experiments demonstrate the effectiveness of our proposed PyramidCLIP. For fair compari- son, when trained on YFCCI5M-V2 (10) dataset, with ResNet50 (1 (11) /VT-B32 (12) /DD/VV--B6 B16 (12) as the image encoder and Transformer as the text encoder, our model achieves the state-of-the of-the-art (SoTA) zero-shot classification on ImageNet (13) with 47.8% 20%6//6.000 /4.00% /5..5% top-1 accuracy. In comparison, the CLIP achieves 37.2% 33.8% /40.7% respectively. Furthermore. when scaling to the large-scale dataset, the results of PyramidCLIP achieve SoTA on several downstream tasks, in particular, the results of PrramidCL.P-ResNeeee trained on 143M image-text pairs surpass that of CLIP trained on 400M data on ImageNet zero-shot classification task, improving the data efficiency of CLIP significantly.
Our main contributions are summarized as follows: (i) We propose PyramidCLIP for more accurate image-text alignment for vision-language model pre-training, which effectively constructs two input pyramids at both sides of the visual encoder and linguistic encoder, and then align the visual elements and linguistic elements via peer-level semantics alignment and cross-level relation alignment. (ii) We soften the loss term of negative samples during the contrast process to ease the strict constraint, so as to alleviate the negative effect caused by local similarities. (ii)) Extensive experiments demonstrate the effectiveness of PyramidCLIP, which achieves SoTA on several downstream tasks.
2 Related Work
Vision-Language Pre-training Vision-Language Pre-training (VLP) learns a strong joint representa- tion between two modalities by pre-training models on large-scale image-text pairs. In terms of the model architecture, the mainstream VI.P models can be divided into two types: single-stream and dual-stream. The former one uses single transformer to model both image and text representations

---

in unified semantic space by concatenating image and text imput embeddingg, including Visual- BERT (2), UNITER (I), UNICODER (4), OSCAR (3) and UNIMO (14). The latter one encodes images and texts separately with decoupled image encoder and text encoder, such as ViL.BERT (15), LXMERT (9), ALIGN (7), CLIP (6), and DeCL.P (10). From a different perspective, the pre-training objective mainly comprises two categories: image-text contrastive learning and masked token tasks based on Language Modeling (LM). Among the methods mentioned above, UNIMO, ALIGN, CLIP and DeCLIP adopt contrastive learning to align the textual and visual representation in unified semantic space, In contrast, VisualBERT, UNITER, LXMERT and UNICODER use masked token tasks, including Masked Language/Region Modeling and autoregressive LM. In this paper, employ dual-stream architecture and contrastive learning for simplicity, flexibility, and relatively cheaper computation cost.
Fine-grained Alignment Due to the semantic gap between image and text, there may be some troubles in directly performing the alignment between these two modalities. For example, some words or phrases in the descriptions may be irrelevant to the images, or the corresponding descrip- tions of the objects in an image may not always be available in the caption. Thus, finer-grained alignment is indispensable, as it can provide more accurate and richer supervision signals of multiple granularity, improving the performance significantly. FILIP (8) improves the contrastive objective to achieve finer-level alignment by using a token-wise maximum similarity between visual and textual tokens. Methods in (3; 16; 17; 18) construct multi-level semantic concepts for finer- grained alignment. OSCAR (3) first introduces multi-level semantics, capturing object region features and the corresponding tags with a pre-trainn trained object detector, then concatenates text, object tags and region features together to learn the joint representations. VI (16) enhances the visual representations of OSCAR by pre-trianing a more powerful object-attribute detector. Both OSCAR and VL form the multi-level semantics only in the visual modality. MVPTR (17) and X-VLM (18) obtain their multi-level semantics concepts in both visual and linguistic modalities. MVPTR limits the interaction between object tags and textual tokens, and learns the object-tag alignment in an explicit manner. It also models the nested property of language by learning phrase-level semantics. X-VLM learns multi-level alignments by positioning vision concepts using given texts, and makes alignments between these two parts. However, in addition the image encoder and the text encoder, the two methods both have an additional cross- encoder, bringing computation overhead.
In this paper, we follow the dual-stream design of CLIP and construct three visual semantics levels and three linguistic semantics levels to form our PyramidCLIP. Different from methods mentioned above, each level is input to the corresponding encoder individually, without concatenating. The obtained three visual representations and three linguistic representations are used to compute six contrastive loss terms, which helps achieve multi-level alignments.
Method
In this section, we introduce the proposed PyramidCLIP for more accurate alignment of image and text for vision-language model pre- training..
3.1 Overall Architecture
The entire framework of the proposed PyramidCLIP is presented in Figure 2. PyramidCLIP is dual-stream network, including a text encoder h and an image encoder f = fz o f1, where fa and fa denote the front part and the rear part of the image encoder respectively. Each encoder consists of a linear projector and a normalization operator in the end, which project the final class token into unified dimension and then normalize it, obtaining the corresponding visual or linguistic representation vector in the same embedded space.
During the training, for each image-text pair (1,T), the image I is transformed into two views. i.e., local view L and global view G, through random crop with different ratios. And text T is input to summary extractor (19) to generate text summarization Ts with higher level semantics. The image global view G and text summarization T's both capture global context information, while the image local view L and original text T contain more detailed information. Therefore, G and T3 regarded as a pair of positive samples, while L and T are regarded as another pair of positive samples, denoted (G, Ts) and TL. These two pairs input to the dual-stream encoder to extract global and local representation pairs, (v",1" and (v', 1²), where 8 = f(G), l' h(Ts), of f(L and
3

---

Visual Modelling
Image Enc. f()
Contrastive Learning
Linguistic Modelling
Summaay Ts
There couch
Summary
View
the living room with two piloows and a bible
Image
12
Enc.
Text T
h((-)
There sewwee couch lhiing room too pink pillows on the couch and a round table from offll.
View
pa
R
Pear-leval Alignment
Object Categories with Artributes ToA
① Las 2 Lus
Object attrioute
Deteclox
Linear Embedding (()
Cross-lavel Alignment
3 Lea 4 Las
Pirk gillow, square pirk pillow, large framed
Lea @ Lgr
hanging painting, gold round brown table.
grewn
Figure Overall architecture of the proposed PrramidCIIP which is dual-stream network. The input elements of visual modelling and linguistic modelling both have three-level semantics. The elements of the two modalities interact through peer-level semantics alignment and cross-level relation alignment.
= h(T). Finally, (v9, 1*) and (v', 1 are pulled together through contrastive learning losses and 2 respectively (refer to Figure 2), and other samples in the same batch are treated as negative samples. We this contrasting process as Peer-level Semantics Alignment.
Furthermore, order to explicitly model the relationship between salient objects in the image, the ROI feature sequence R = {01, 02, .... 0s } of M detected salient objects in the image I, with the category and attribute information for each object, are extracted through a pre-trained object-attribute detector. Then linear embedding module yp is used to transform R into the latent dimension of Multi-head Self-attention (MHSA) layer (12) in the image encoder. The feature sequence is successively fed into the rear part f2. which contains capture the relation between these salient instances, generating the final representation vector v", i..,, D" (R)) For the object categories with attributes, we join them together, constructing another text description TOA. to provide more granular, comprehensive and accurate caption for the image. Then we feed it into the text encoder, generating the representation vector l h(ToA). To avoid the relation modelling being overwhelmed by the context semantic modelling, we have (w. 19), (v", 1²), (w', Id) and (v", 1`) as another four positive pairs, and the distances between which are narrowed through contrastive learning losses Cross-level Relation Alignment.
MHSA layers, adaptively
and respectively, termed
It is worth noting that at the inference stage, only the original image-text pair (1,T") used, i.e., the visual representation e² from and the linguistic representation 1' from I".
3.2 Peer-level Semantics Alignment
Now we present the details the peer-level semanties alignment. mentioned above, the dual- stream vision-language contrastive learning methods such CLIP strongly rely on the assumption that image-text pairs are good quality of one-to-one correspondence. However, semantie mismatch between images and text captions often occurs in the automatically harvested data. Therefore, we construct an input pyramid with multi-level semantics on both sides of the dual-stream network, and then align image and text within the same semantics level. Specifically, the image / is transformed to the global view G and the local view L through two random crops with different ratios. For the text caption, besides the original caption I', text summarization T's with more compact semantics is extracted using pre-trained text summarization extractor.
Coarse-grained Global Contrast We set the random crop ratio for generating global view G to be [0.9, 1], which basically contains the information the original image. Text summarization Tş condenses the original caption T. removing some redundant and overly detailed information in the T. G and Ts both capture global information and can be used as pairs of positive samples. The projected embedding d and l of G and T's are pulled closer through contrastive learning.

---

Fine-grained Local Contrast Since the alignment of the global view G with the text summarization Ts described above is relatively coarse, finer- finer-grained information is largely discarded. Intuitively, some image sub-regions can be aligned with the original caption T. To this end, we introduce fine-grained local contrast. We set the random cropping ratio for generating local view I. to be [0.5, I]. which focuses on the sub-region of the image T. Then the projected embeddings o' and 1 of L and T are also brought together through contrastive loss (refer to Section 3.4).
Naturally, we have also tried to bring the finer-grained, peer-level (v", 19) closer, but there is no further gain (see Appendix F).
3.3 Cross-level Relation Alignment
To further improve the alignment precision, we introduce the ROI feature of each salient object in the image and the corresponding description with category and attributes, using pre-trained object- attribute detector, as the most fine-grained semantic level to provide more accurate supervisions. Specifically, given an image with M salient objects, the extracted visual semanties of each object region is formulated as Io'm ]. where m denotes the mihh object, o'm is a 1948-dimensional feature vector and m is 4-dimensional normalized position vector indicating the coordinates of top-left and bottom-right corners. By concatenating o'm and Em. we have the 2052-dimensional position-sensittee ROI feature vector Om forming the ROI feature sequence R- {0r, 02, ..... OM with the order organized from high confidence to low. Then R E IR x2052 is transformed into R R. Mxd using the projector in the embedding module yp, where indicates the latent dimension of the MHSA layers in the image encoder. A randomly initialized d-dimensional class token is additionally appended at the front, resulting I E TR. (1+ which is further feed into the rear part f2 of the image encoder to compute the normalized ROI relation embedding v", i.e., = (R) and v" = f2()). Note that feature sequence F is position-sensitin before enters into the MHSA layer. Meanwhile, the detected names of category and attributes for each salient object form phrase with one or more adjectives (attributes) modifying noun (category), like "gold round brown table" in Figure Then all the M phrases from M salient objects are joint into text description TOA with the same order as the ROI feature sequence, and the phrases are separated by commas. Next, TOA is input to the text encoder to obtain the embedding 1 = h(ToA).
following R. Thus, positional embedding is not applied
To enhance the relation modelling capacity of the text encoder, while avoiding weakening the reasoning ability of the image encoder, (v", 1²), (v", 1²), (v', 19) and (v", I') are used as another four positive pairs, and the distance between each pair is minimized simultaneously. Since the object-level inputs R and Toa are direct concatenation of feature vectors and phrases respectively, hence very fine-gre -he-graided, while other inputs G, L, Ts and T are complete images or sentences, we term this contrasting process cross-level relation alignment.
In the case that the visual model is a con- volution neural network (CNN), the tradi- tional pooling layer is replaced by attention pooling, which actually a MHSA layer. So the embedded ROI feature sequence FF input to the attention pooling layer, i.e., fa. which indicates the final attention pool- ing layer, as shown in Figure 3(a). For the transformer-based visual model (ViT), the sequence F can be directly input to trans- former layer. Considering that F already encodes high-level visual semantics, we feed it into the rear part f2 of the ViT en- coder, see Figure (b))
Attention Pooling
f2
Transformer Layers
Ingge
Deprivsaie
Ing2Tokens
Transformer Layers with LeFF
x1,
Tolessding
f1()
CNN Backbone
Img2Takens
Conv
L (n) CWWN-booed Image Encoder
G
[b) VIT-based Image Encoder
(4) LoFF
Figure 3: (a) The schematic of CNN-based image encoder. (b) The schematic of ViT-based image encoder. (c) The struc- ture LcFF module in VIT.
Besides, as pointed out in (20; 21), the standard ViT may not fully leverage the local context informa- tion, which limits the visual representation capacity of ViT-based image encoder. Following (21), we incorporate depth-wiss wise convolution into the Feed-Forward module of ViT, termed Locally-enhaneee Feed-Forward (LeFF), improving the leteh-level local perception and interaction. The structure of LeFF is shown in Figure 3(c). First, the patch tokens are projected into higher dimension through a linear projection layer and reshaped. Next, 3 x 3 depth-wise convolution is utilized to capture the local information. Then the feature maps are flattened to token sequence and re-projected into
5

---

the initial dimension. While the class token is identical during the process and is concatenated with locally-enn eocallyenhannee patch tokens, generating the final output. As depicted in Figure 3(b), LeFF is only applied in the front part fi of the ViT-based image encoder, since it is clearly not suitable for the embedded ROI feature sequence F. In Figure 3(b), denotes the total number of transformer layers in VIT and L is the number of transformer layers with LeFF in f1- And I. = 12 and I,, 9 in the experiments. The influence of different settings of L. can be seen in Appendix F.
3.4 Softened Objective Function
For a batch of N image-text pairs {(Ti, Ti) where indicates the ieh pair, the normalized em- bedded vectors {uf, oi of 42 [ of the same dimension are obtained by the dual-stream encoders. In this formulation, vb. o and of are generated by the image encoder from global-crop image G, local-crop image L and ROI feature sequence R respectively, while 47, I and 1 are generated by the text encoder from text summarization Ts, original text T and object-attribute description ToA respectively. Then
this vector group to construct six supervision signals LGs, LLT. LGA. LRS, LLA and LRT for in-batch contrastive learning, which can be calculated with {(v] [r);(+))==_l - [-))) ,())) ()))
19) and {(wf, ((vf, ) respec- tively. Our six contrastive losses, with the formulation of InfoNCE (22), are designed to achieve the alignment between visual representation and linguistic representation from disparate seman- tic levels. Take the first loss term LGs from {( 1)) as an example. For the ieh pair, the normalized vision-to-lan -a-nuuuge similarity p (G) = {pl (G) and language-to- tostugge-ttvviiine vision similarity PT(Ts) = (rly (Ts can be calculated through:
exp(sim(vf .1)/T)
exp(sim (L). e///TT)
(Ts) = Σ exp(sim w/)/T)²
(1)
exp(sim(o] 43)/>*
where is learnable temperature parameter initialized with 0.07 and the function sim(.) conducts dot product to measure the similarity scores.
In standard practice, for the deh pair, the corresponding one-hot label vectors of the ground-truth yf (G) = {uy [ and (T's) = [uly (1 with positive pair denoted by I and negatives by O, are used as the targets to calculate cross-entropy. This kind of hard targets assumes there absolutely no similarity between unpaired image and text. However, within large-size batch, unpaired image and text may have more less local similarities, ie.. some local regions in image may be matched with some words or phrases in other unpaired texts. To address this problem for better generalization, targets 7.' (G) and (Tss ) for the ith pair can be formulated as:
use label smoothing soften the hard targets. The corresponding softened
O (G) = (1 a)v (G) + a/(N 1), (Ts) = (1 - (Ts) + a/(N 1),
(2)
where a is the smoothing hyper-paramette set to 0.2 our experiments. Then Las can be written as:
N
N
1 LGs = 2N
ΣΣ 1=1==1
Co (G) log(p))(G) + (Ts) log(r)y(Ts)) (Ts))).
The other loss terms LLT, LGA. LRS. LLA and RT can be calculated similarly. We then divide them into three groups that are respectively the peer-level alignment &peer = (Los + LLT) the global-relation cross-level alignment Lglobal = (&Ga + ass//2 and the local-relation cross-level cross alignment L Clocal = (CLA + LRT) Therefore, the overall objective function of PyramidCLIP is: cross
L = (1 X - Sperr + Algohall + juClocal
(4)
where the loss weights À and are both set to 1/3 in our experiments.
Experiments
4.1 Implementation Details and Datasets
Pre-training Stage We experiment on three different architectures, ByamidCLLP-RRsNNesovvY B32/ViT-B16 16, according to the choice of image encoder. Their detailed architecture designs follow that of CLIP (6). LAION99M contains 99M image-text pairs with the highest similarity selected from
LAION400M (23) according to the similarity scores provided by the producer. We use the publicly

---

released T5 model (19) to extract text summarization for all texts and utilize an object-attribute detector pre-trained by VinVL (16) extract salient object features together with category and attribute information in the image. Please refer to Appendix A for details.
Table 1: Pre-training datasets.
Dataset
Size
SBU (24)
IM
CC3M (25)
3M
Downstream Tasks We validate the effectiveness of our proposed method on five downstream tasks: zero-shot image classification, zero-shot image-text retrieval, linear probe, object detection and instance segmentation. For classification, experiments are carried out on 11 datasets, such as ImageNet (13), CIFAR- 100 (28). For image-text text retrieval, experiments are conducted on Flickr30K (29) and MS- AOSC (30). For object detection and instance segmentation, the proposed method is verified
CC12M (26)
10M
YFCC15M-V V1 (27)
15M
YFCC15M-V2 (10)
15M
LAION99M (23)
99M
Total
143M
on PASCAL VOC (31) and MS-COCO. More details can be found in Appendix B.
Table 2: Zero-shot(ZS) classification results on ImageNet.
4.2 Fair Comparison against SoTA
We first compare our method against other SoTA ap- proaches on ImageNet zero-shot classification task using the same amount of pre-training data YFCC15M-V V2 and the results are shown in Table 2. It can be seen that com- pared to CLIP, PyramidCLIP improves the top-1 accu- racy by 10.6% / 13.2% 13.2% 10.0% when the visual model is ResNet50/ VTT-B32 B32///T-B/BB VVTT-B16 respectively. Furthermore, PyramidCLIP outperforms all other So'T approaches by a large margin. In addition, since the distribution of different datasets can vary vastly, also conduct ex- periments on YFCC15M-VI and LA, which is obtained by sampling 15 million image-text pairs from LAION99M for fair comparison. The results can be seen in Appendix C and our PyramidCLIP still shows great superiority.
Image Encoder
ImageNet ZS Top1
Method
CLIP (6)
37.2
SLIP (32)
28.51
FILIP (8)
ResNet50
21.3
DECLIP (10)
44.41
PyramidCLIP
47.8
CLIP (6)
32.81
SLIP(32)
34.3*
FILIP (8)
VIT-B/32
39.5t
DECLIP (10)
43.21
DeFILIP (33)
45.0"
PyramidCLIP
46.0
CLIP
40.7
PyramidCLIP ViT-B/16
50.7
4.3 Comparison on Large-scale Datasets
Our Implementation
Reported in (33)
We further validate the effectiveness of our method on a large-scale dataset, i.e., 143M image-text pairs, and downstream zero-shot image-text retrieval and image classification results are shown in Table 3. It can be seen that on image-text retrieval task, PyramidCLIP exceeds CLIP trained on 400M data and DECLIP by large margin. And on ImageNet classification task, with the same amount of pre-training data, PyramidCLIP significantly exceeds the results of CLIP by 6. 6.1%/3.8%%/. using B.NNNc550//VT-B3//VVT 16 as image encoder. Furthermore, it is worth noting that, when the vision model is ResNet50, PyramidCLIP trained on 143M data surpass CLIP using 400M data, which greatly improves data efficiency.
4.4 Transferability to Small-scale Classification Datasets
In this section, we validate the transferability of our method on 10 relatively small downstream classification datasets, both on zero-shot and linear probe tasks. The results are shown Table 4. can be seen that the average accuracy of PyramidCIIIP on 10 datasets all exceed CLIP trained on 400M data on two kinds of tasks. It is worth noting that our pre-training data is less than 36% of CLIP, but the average accuracy is better, indicating higher data utilization.
4.5 Transferability to Object Detection and Instance Segmentation
In order to verify that our model can better exploit the relationship between objects in the image, we further validate our models on object detection and instance segmentation tasks. Specifically, we take the visual model ResNer50 to initialize the backbone of Faster R-CNN (34) and Mask R-CNN (35) and then all the parameters are fine-tuned. The results are shown the Table 6. It can be seen that our model significantly outperforms random initialization and surpasses CLIP and DECLIP model.
7

---

Table Zero-shot image-ter text retrieval results and image classification top-1 accuracy. IN denotes ImageNet.
FOickr30K
MS-COCO
Image Encoder
Pretrain Dataset
IN ZS Top1
Method
12T
T2I
12T
R@5
R@
R@5
R@
R@5
R@
R@5
CLIP
400M
79.2
95.2
57.9
84.1
47.6
73.1
27.4
51.8
59.6
DECLIPÍ
ResNet50
88M
60.4
85.3
46.3
74.4
32.0
57.8
21.7
44.6
62.5
CLIP
143M
80.6
95.7
63.6
87.3
51.8
76.4
34.0
60.0
55.3
PyramidCLIP
143M
86.3
98.0
71.6
91.3
55.0
79.8
39.6
66.2
61.4
CLIP
400M
77.6
93.6
59.0
83.7
49.2
74.1
29.8
54.4
63.2
DECLIP
VIT-B/32
88M
59.8
84.4
46.2
74.5
32.6
59.1
22.1
45.8
66.6
CLIP
143M
81.3
95.4
63.3
87.0
51.1
76.4
34.4
60.6
58.0
PyramidCLIP
143M
84.2
96.4
69.1
89.8
52.8
78.1
38.8
64.9
61.8
CLIP
400M
84.6
97.3
65.0
87.8
51.7
76.1
32.5
57.7
68.8
CLIP
ViT-B/16
143M
84.5
97.4
70.5
90.9
56.9
79.6
38.8
65.0
63.4
PyramidCLIP
143M
97.7
74.5
55.7
42.6
66.9
Tested with the released model: httpss///gitub..cmoooppaaaiCCCC
Our Implementation
Tested with:
Table 4: Accuracy on datasets with ResNet50 image encoder. C10/100/F101/FLL0///
CIFAR-
AVG represents average accuracy across 10 datasets.
Pretrain Dataset
Task
Method
PETS C10 C100 DTD CARS F101 FLOW
AIR SUN
CAL
AVG
CLIP
400M
85.4
75.6
41.6
41.7
55.8
81.1
65.9
19.3
59.6
82.1
60.8
Zero Shot
CLIPP
143M
77.0
56.4
26.7
41.4
54.6
69.8
60.4
7.3
60.6
76.1
53.6
PyramidCLIP
143M
83.7
81.5
53.7
47.2
65.0
67.8
65.8
12.6
65.8
81.7
62.4
CLIP
400M
85.1
88.7
70.3
76.4
78.3
86.4
96.1
49.1
73.3
89.6
79.3
Linear Probe
DECLIP
88M
88.7
89.8
71.2
76.8
81.7
82.7
99.2
48.4
72.8
93.9
80.5
CLIP
143M
82.9
84.5
64.5
74.3
79.3
80.5
93.2
45.4
74.7
91.5
77.1
PyramidCLIP
143M
87.8
91.8
75.6
75.8
81.8
81.9
93.0
53.1
76.1
94.2
81.3
Our Implementation
4.6
Ablation Study
In this section, verify the effectiveness of each component PyramidCLIP on downstream zero- shot ImageNet classification task, and all the experi- ments are pre-trained for epochs on YFCC15M-VI. The results listed in Table 5, which indicate that on the basis of the peer-level alignment, all the other components including cross-level global-relation and local-relation alignment, LeFF in ViT and softened objectives can bring significant gains individually. More ablation results can be seen in Appendix F, including the detailed ablation on L,, the choice of peer-level alignment and the effectiveness each com- ponent on downstream object detection task.
Table Ablation study component on ImageNet zero-shot classification task. "Soften" means all the objectives are softened.
Image Encoder
Components
ImageNet ZS Topl
Upeer & globall L glecal Soften LeFF
cecoe croos
32.8
35.0(+2.2)
ResNer50
36.7(+3..9
38.6(+5.8)
28.8
32. 1(+3.)
VIT-R-32
33.4(+4.6)
35.0 35.0(+6.2)
35.9(+7.1)
4.7 Visualization
Semantic Features We utilize t-SNE (36) to visualize the learned semantic features of CIFAR-10 (28).
Each text feature of 10 categories obtained using the ensemble of 80 prompt templates. And image features are extracted with ResNet50 visual encoder. As depicted in Figure CLIP pre-trained on 143M data has poor clustering performance, with the visual features of most categories overlapping heavily and the textual features of some two categories being very close, such as (automobile. truck) and (frog, deer). Although CLIP pre-trained on 400M data (6) performs better than CLIP on 143M data, the visual features of some categories like dog and cat still have large overlap. In comparison, the semantic features extracted by PyramidCIIIP on 143M data, are well separated with each textual feature surrounded by most visual features of the same category.

---

Table 6: Object detection and instance segmentation results VOC and COCO with ResNet50 backbone.
Object Detection
Instance Segmentation
Initialized Weights
Pre-train Dataset
VOC
COCO
COCO
pbb
75
phi
50
A AP
A
50
76
Random CLIP (6)
26.5
51.6
22.8
28.5
46.2
29.8
25.6
43.4
26.8
400M
45.5
73.5
47.9
36.5
56.1
38.8
31.9
52.7
33.5
DECLIP (10)
88M
50.0
77.4
53.6
37.4
57.2
40.1
32.5
53.6
34.4
CIIP
143M
50.9
78.7
54.7
38.0
58.1
40.8
33.0
54.6
35.1
(a) CLIP on 143M data
(b) CLIP on 400M data (6)
(c) PyramidCLIP on 143M data
Figure Visualization of semantic features of CIFAR- 10 test Big points represent text features and small points indicate image features. Different colors represent different categories.
Grad-CAM Heatmaps Grad-CAM (37) is also utilized to help understand why PyramidCLIP outperforms CLIP. Specifically, we conduct this through text-to-image retrieval on MS-COCO, using LLlPPPranidCCIP-ReSII/
pre-trained on 143M data. For each query text, the model retrieves top5 images with highest similarities. Then for each of the five retrieved images, we use Grad-CAM to find which areas have the highest activation to the query text. As shown in Figure 5, PyramidCLIP has better retrieval performance than CLIP. Moreover, PyramidCLIP can accurately capture the complete object regions that are highly matched with the query texts, while CLIP captures regions with either components missing or additional noises, or even obtains completely irrelevant matches. More visualization results can be found in Appendix G.
Query: There middle busy road.
Query An sitting bench next fence.
CLIP
PyramidCLIP
Figure 5: Grad-CAM heatmaps for top5 retrieved images. From left to right are images from rank1 to rank5. Red box indicates the groundtruth image matched with the query text.
Conclusion
In this paper, we have proposed hierarchical pre-training method, termed PyramidCLIP, to achieve improved alignment between visual and linguistic modalities. resolves the issue that the webly- crawled data is not in perfect one-to-one correspondence by explicitly constructing pyramidal semantic inputs at the both sides of dual-stream network. We also show that softened peer-level semantics alignment and cross-level relation alignment can interact between two modalities and are beneficial. PyramidCIIPP achieves the state-of-the-a results on five downstream tasks, which demonstrates the superiority.

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [No]
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work
does not have any potential negative societal impacts.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either the supplemental material or as a URL)? [No] Our codes may be released in the future.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A.
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [No] We did not run the same experiment for multiple times.
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Appendix A.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material or as a URL? [No]
Our codes may be released in the future.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No] Existing assets used in our work are all publicly released. (e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [No] Datasets used in our work are all publicly released without this problem.
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
References
Y.-C. Chen, L. Li, L. Yu, A. EI Kholy, F. Ahmed, Z. Gan, Y. Cheng, and J. Liu, "Uniter: Universal image-text representation leamiing"' in Proc. European Conf. Computer Vision, 2020, pp. 104-120.
[2] L. H. Li, M. Yatskar, D. Yin, C.-J. Hsich, and K.W. Chang, "Visualbert: A simple and performant baseline for vision and language," arXiv preprint arXiv: 198, 03557, 2019.
[3] X. Li, X. Yin, C. Li, P. Zhang, X. Hu, I.. Zhang, L. Wang, H. Hu, I.. Dong, F. Wei et al., "Oscar: Object- semantics aligned pre-training for vision-language tasks," in Proc. European Conf. Computer Vision, 2020, pp. 121-137.
[4] G. Li, N. Duan, Y. Fang, M. Gong, and D. Jiang, "Unicoder-vl: A universal encoder for vision and language by cross-modal pre- treiniaa in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, 07, 2020, pp. 11 336 344.
10

---

[5] W. Su, X. Zha, Y. Cao. B. L., Lu, F. Wei, and J. Dai, "VI-bert: Pre-training of generic visual-linguistic representations." arXiv preprint arXiv:1908.00 198.0050000 2019.
[6] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark et al., "Learning transferable visual models from natural language supervision," " Proc. Interna- tional Conf. Machine Learning, 2021, pp. 8748-8763.
[7] C. Jia, Y. Yang, Y. Xia, Y.TT. Chen, Z. Parekh, H. Pham, Q. Lc, Y.H. Sung, Z. Li, and T. Duerig, "Scaling up visual and vision-language representation learning with noisy text supervision," in International Conference on Machine Learning. PMLR, 2021, pp. 4904-4916.
[8] L. Yao, R. Huang. L. Hou, G. Lu, M. Niu, H. Xu, X. Liang. Z. Li, X. Jiang, and C. Xu, "Filip: Fine- Ene-grained interactive language-image pre- wre-training," arXiv preprint arXiv:2111.0778 2021.
[9] H. Tan and M. Bansal, "Lxmert: Learning cross-modality encoder representations from transformers." arXiv preprint arXiv: 190.9999880990.. 2019.
[10] Y. Li, F. Liang, L. Zhao, Y. Cui, W. Ouyang, J. Shao, F. Yu, and J. Yan, "Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm." arXiv preprint arXiv:21 2021.
[11] K. He, Zhang. S. Ren, and Sun, "Deep residual learning for image recognition." in Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2016, pp. 770-778.
[12] A. Dosovitskiy, L.. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly et al., "An image is worth 16x16 16 words: Transformers for image recognition at scale," arXiv preprint arXiv:2010. 19299, 2020.
[13] J. Deng, W. Dong, R. Socher, L..J. Li, K. Li, and L.. Fei-Fei, "Imagenet: A large-scale hicrarchical image database." " Proc. IEEE Conf. Computer Vision and Puttern Recognition, 2009. pp. 248-255.
[14] W. Li, C. Gao, G. Niu, X. Xiao, H. Liu, J. Liu, H. Wu, and H. Wang, "Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning," arXiv preprint arXiv:2012. 2020.
[15] J. Lu, D. Batra, D. Parikh, and S. Lee, "Vilbert: Pretraining task agnostic visiolinguistic representations for lannnuae tasks," " Proc. Advances Neural Information Processing Systems, vol. 32, 2019.
[16] P. Zhang. X. Li, Hu, Yang, L. Zhang, L. Wang, Choi. and J. Gao, "Vinvl: Revisiting visual repre- sentations in vision-language models," in Proc. IEEE/CVF Conf. Computer Vision & Pattern Recognition, 2021, pp. 5579-5588.
[17] Z. Li, Z. Fan, H. Tou, Z. Wei, and X. Huang. "Myptr: Multi-stage vision-language pre-training via lulti- semantic alignment," arXiv preprint arXiv:2201.12596 2022.
[18] Y. Zeng, X. Zhang, and H. Li, "Multi-grained vision language pre-training: Aligning texts with visual concepts." " arXiv preprint arXiv:2111.08277 2021.
[19] C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu, "Exploring the limits of transfer learning with unified text-to-text transformer." arXiv preprint arXiv: 10683, 2019.
[20] J. Guo, K. Han, H. Wu, C. Xu, Y. Tang, C. Xu, and Y. Wang, "Cmt: Convolutional neural networks meet vision transformers," arXiv preprint arXiv:2107.06263 2021.
[21] K. Yuan, Guo, Z. Liu, A. Zhou, Yu, and W. Wu, "Incorporating convolution designs into visual transformers," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 779-588.
[22] A. Van den Oord, Li, O. Vinyals et al., "Representation learning with contrastive predictive coding." arXiv preprint arXivv1807.03748. vol. 2, no. 3. p. 4, 2018.
[23] C. Schuhmann, R. Vencu, R. Beaumont, Kaczmarezyk, C. Mullis, A. Katta, Coombes, J. Jitsev, and
A. Komatsuzaki, "Laion-400m: Open datasct of clip-filtered 400 million image-text pairs," arXiv preprint arXiv:2111.0211 2021.
[24] V. Ordonez, G. Kulkarni, and T. Berg, "Im2text: Describing images using I million captioned photographs." Proc. Advances in Neural Information Processing Systems, vol. 24, 2011.
11

---

[25] P. Sharma, N. Ding, S. Goodman, and R. Soricut, "Conceptual captions: A cleaned. hypemyyed. image alt- text dataset for automatic image captioning." in Proceedings the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), 2018, pp. 2556-2565.
[26] S. Changpinyo, P. Sharma, N. Ding, and R. Soricut, "Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 3558-3566.
[27] B. Thomee, D. Shamma, G. Friedland, B. Elizalde, K. Poland, Borth, and L.-J. 100m The new data in multimedia research," Communications of the ACM, vol. 59, no. 2, pp. 43, 2016.
[28] Krizhevsky, G. Hinton el al., "Learning multiple layers of features from tiny images." " Tech. Rep.. 2009.
[29] M. Hodosh, Young, and J. Hockenmaier, "Framing image description ranking task: Data, models and evaluation metries," Journal of Artificial Intelligence Research, vol. pp. 853 899.8999.00 2013.
[30] T.-Y. Lin, M. Maire, S. Belongie, Hays, Perona, D. Ramanan, P. Dollár, and C. L. Zitnick, "Microsoft Common objects context," " Proc. European Conf. Computer Vixion, 2014, pp. 740-755.
CocO:
[31] M. Everingham, S. Eslami, I.. Van Gool, C. K. Williams, J. Winn, and A. Zisserman, "The pascal visual object classes challenge: A retrospective." International journal of computer vision, vol. 111, no. pp.
98-136, 2015.
[32] N. Mu, Kirillov, Wagner, and Xie, "Slip: Self-supervision meets language-image pre-training." " arXiv preprint arXiv:2112.11 12..2500, 2021.
[33] Y. Cui, L. Zhao, Liang, Li, Shao, "Democratizing contrastive sanguuge--maae pre-training: clip benchmark of data, model, and supervision," arXiv preprini arXiv:2203.05796, 2022.
[34] S. Ren, K. He, R. Girshick, and Sun, "Faster R-CNN: Towards real-time object detection with region proposal networks," Proc. Advances in Neural Information Processing Systems, vol. 28, 2015.
[35] K. He, G. Gkioxari, P. Dollár, and R. Girshick, "Mask r-cnn," Proceedings of the IEEE international conference on computer vision, 2017, pp. 2961-2969.
[36] L. van der Maaten and G. Hinton, "Viualizing data using t-sne," Journal of Machine Learning Research, 2579-2605, 2008.
[37] R. R. Selvaraju, M. Cogswell, A. Das. R. Vedantam, D. Parikh, D. Batra, "Grad-cam: Visual explanations from deep networks via gradient-based localization," in Proceedings of the IEEE international conference computer vision, 2017, p. 618-626.
12


Reviews:
[{"review_id": "npDczYLTMFd", "reviewer": "Reviewer_e9Ef", "summary": "This work introduces to construct an input pyramid with different semantic levels for each modality. It then aligns visual elements and linguistic elements in a hierarchical way. The proposed PyramidCLIP outperforms CLIP with a large margin. \n\n", "questions": "Please see my comments in the weaknesses part.", "limitations": "Yes.", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- The results are very promising. The model outperforms the sota methods across many datasets.\n- The overall framework is easy to follow.\n\nWeakness:\n- The idea of introducing ROI features may increase the computational cost significantly. \n- In peer-level semantics alignment, the authors introduced coarse-grained global contrast learning and fine-grained local contrast learning. However, I didn’t find the more studies to validate that combining two contrast alignments helps model training.\n- The authors described the cross-level relation alignment in Sec 3.3. Three structures are proposed but I suggest the authors providing more analysis to prove their effectiveness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "QIC4i5R3sUc", "reviewer": "Reviewer_BusK", "summary": "Under the contrastive CLIP learning framework, the work proposes to utilize more fine-grained information to produce multiple views of both the image and text during training, and hence constructs more contrastive loss terms across different views. During inference/evaluation, only the standard view is used.\n\nEmpirically, with 3 different architectures (ResNet50/ViT-B32/ViT-B16), different pretraining data scales and several down-stream datasets, authors show that the proposed approach achieves clear gain over the baseline systems. ", "questions": "If my understanding above is correct, how would the comparison look like if similar training cost is invested in training the baselines, particularly with soften and LeFF disabled? (NOTE: In my opinion, this is very critical for judging whether the proposed method is really useful. Hence, my final review will largely depend on this.)", "limitations": "I don't see any particular problem here.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The work follows a natural motivation and achieves very good empirical gain over some strong baseline systems. Overall, the paper is well written and the empirical study is solid.\n\nA key concern I have is whether the comparison is fair enough. If I understand correctly, for each view of either the image or the text, we need to feed the view into the model once, which leads to roughly 2x - 3x additional computation cost. Again, if my understanding is correct, despite using the same batch size, the actual pretraining cost might be much higher for the proposed method than baselines, making the comparison less information. A better comparison seems to be make the batch size of baselines larger until the training cost is comparable.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "K2IfD7o8HY", "reviewer": "Reviewer_QuL5", "summary": "This paper proposes a new framework called PyramidCLIP for vision-language pre-training (VLP). PyramidCLIP first extracts multi-level features from both the visual and linguistic domains and then conducts contrastive learning by aligning visual and linguistic features in both peer-level and cross-level ways. In this way, the vision and language representations learned by PyramidCLIP encode better image-text alignment, which alleviates the semantic mismatch problem that exists in the image-text pairs for pre-training. Moreover, the authors also replace the loss term of the negative samples in contrastive learning with a softened version, to tackle the problem that different image-text pairs may have potential correlation. The empirical results show that PyramidCLIP clearly outperforms the CLIP baseline in a variety of downstream tasks and also achieves SoTA performance on several downstream tasks, as compared with other VLP models.", "questions": "* Could you provide an explanation about how the cross-level alignment helps the modelling of relations between salient objects?\n* Could provide a more detailed discussion on the difference between PyramidCLIP and the VLP methods that also introduce multi-level semantics?\n* Why not compare PyramidCLIP with MVPTR and X-VLM?", "limitations": "* The soften objective function treats all the negative samples equally using label smoothing, which is suboptimal considering different image-text pairs should have different degrees of correlation.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n* This paper addresses the problem of the quality of image-text pairs (which is an important problem in VLP) using hierarchical feature alignment. The core idea and specific designs of the proposed PyramidCLIP are generally reasonable.\n* The authors conduct extensive experiments to demonstrate the effectiveness of PyramidCLIP, covering different backbone architectures (ResNet50 and ViT-B), pre-training datasets of varying sizes and a variety of downstream tasks (zero-shot image classification, zero-shot image-text retrieval, linear probe, object detection and instance segmentation).\n* The ablation study is comprehensive, showing that all the components of the proposed method are conducive.\n* Qualitative analyses are conducted to intuitively show that PyramidCLIP learns better vision and linguistic representation than CLIP.\n* The paper is generally well-written and easy to follow.\n\n\n### Weaknesses\n\n1. Some specific designs of PyramidCLIP seem counter-intuitive.\n* Random crop cannot guarantee the quality, especially for the local view. For example, if the local view is irrelevant to the textual description, minimizing the distance of the corresponding features may confuse the model.\n* It is unclear how the cross-level alignment helps the modelling of relations between salient objects. For example, how does the model know that \"a table is next to a chair\", by contrasting the ROI features to the textual summary or the original text? It is better to provide an intuitive explanation.\n\n2. The difference between PyramidCLIP and existing VLP methods that also introduce multi-level semantics is not clearly discussed. In the related work, the authors state that \"Different from methods mentioned above, each level is input to the corresponding encoder individually, without concatenating.\" Such a difference seems trivial and incremental.\n\n3. Many methods described in the related work are not compared in the experiments, especially the methods that also introduce multi-level semantics (e.g., MVPTR and X-VLM).\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ib-op6cQ7lM", "reviewer": "Reviewer_jqKi", "summary": "This paper proposes to improve CLIP by adding contrastive losses at different levels. Specifically, for each image in the original data, they construct image views at global, local, and region levels, and also create their corresponding text captions. Therefore, for each image-caption pair in the original dataset, they can create two additional image-caption pairs and the model is trained with additional contrastive losses given the newly constructed data. They also soften the original contrastive loss with a label-smoothing-like technique. Experiments demonstrate improvements over several baselines on zero-shot retrieval, linear probing on image classification,  object detection, and semantic segmentation tasks.", "questions": "Would it make more sense to use a keyword extraction model than a summarization model?", "limitations": "I am not sure if their method can be scalable as when more data is included, many of the problems mentioned in the paper may be alleviated.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The empirical results are strong as they can outperform several popular baselines.\n2. The paper is well-structured.\n3. I like the idea of softening the contrastive loss objective, which makes sense and the paper shows it works well.\n\nWeaknesses:\n1. The comparisons between their model and baselines may not be fair. The paper uses multiple pre-trained models for their model (e.g. text summarization and object detection models) which can introduce more supervision. Also, because their method will create more image-caption pairs for their model, they would have more pretraining data than the baselines.\n2. Some designs are questionable. For example, the text captions in image-caption datasets are typically short, whereas they use a pre-trained text summarization model, which is designed for summarizing long documents, to further shorten the captions, which does not make sense to me.\n3. While it is true that the pre-training datasets can be noisy, the problem may be alleviated if sufficiently large data are used. It would be good to see if they can keep the performance gain as more data are included. For example, they can plot performance-data size curves for both their model and baselines and see the performance gap when the data size varies.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Ch9vFhjwDMg", "reviewer": "Reviewer_WsHx", "summary": "This paper proposes hierarchical feature alignment for vision language pre-training, called PyramidCLIP, which alleviates semantic mismatch as well as mutual compatibility problems, i.e. false positives and false negatives. PyramidCLIP constructs inputs with three levels of semantics in visual and language modalities respectively and then resolves semantic mismatch through peer-level semantic alignment and cross-level relation alignment. In addition, PyramidCLIP adopts a soft form of InfoNCE to deal with mutual compatibility.", "questions": "See the weaknesses.", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. This paper is well written and easy to follow.\n\n2. The motivation and solution of the article are clear. More precise hierarchical feature alignment for tackling semantic mismatch, and softening InfoNCE for mutual compatibility.\n\n3. The experiments are well designed, and the results are excellent.\n\n\nLimitations:\n\nHave the authors adequately addressed the limitations and potential negative social impact of their work? If not, please include constructive suggestions for improvement. Authors should be rewarded rather than punished for being upfront about the limitations of their work and any potential negative societal impact.\n\n1. Needs more explanation about how the training set is constructed.\n\n2. In order to compare with some recently published works(e.g. [1]), it is recommended that the author can supplement the results on smaller scale datasets, such as cc3m.\n\n3. The categories obtained by object detection are simply joined by commas. Whether different joint forms have an impact on the results (e.g. splicing with spaces)? \n\n[1] Robust Cross-Modal Representation Learning with Progressive Self-Distillation. CVPR 2022.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept