FILIP: Fine-grained Interactive Language-Image Pre-Training
Authors: Lewei Yao
Runhui Huang
Lu Hou
Guansong Lu
Minzhe Niu
Hang Xu
Xiaodan Liang
Zhenguo Li
Xin Jiang
Chunjing Xu
Venue: ICLR 2022 Poster (2022)
Keywords: Visual-language pretraining
Language-Image Pretraining
Multi-modality model

Abstract: Unsupervised large-scale vision-language pre-training has shown promising advances on various downstream tasks. Existing methods often model the cross-modal interaction either via the similarity of the global feature of each modality which misses sufficient information, or finer-grained interactions using cross/self-attention upon visual and textual tokens. However, cross/self-attention suffers from inferior efficiency in both training and inference. In this paper, we introduce a large-scale Fine-grained Interactive Language-Image Pre-training (FILIP) to achieve finer-level alignment through a cross-modal late interaction mechanism, which uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective. FILIP successfully leverages the finer-grained expressiveness between image patches and textual words by modifying only contrastive loss, while simultaneously gaining the ability to pre-compute image and text representations offline at inference, keeping both large-scale training and inference efficient. Furthermore, we construct a new large-scale image-text pair dataset called FILIP300M for pre-training. Experiments show that FILIP achieves state-of-the-art performance on multiple downstream vision-language tasks including zero-shot image classification and image-text retrieval. The visualization on word-patch alignment further shows that FILIP can learn meaningful fine-grained features with promising localization ability.

Published as a conference paper at ICLR 2022
FILIP: FINE-GRAINED INTERACTIVE LANGUAGE-
IMAGE
Lewei Yao 1,2+ Runhui Huang Hang Xu Xiaodan Liang 3f
Lu Hou
Guansong Lu
Minzhe Niu
Zhenguo Li
Xin Jiang'
Chunjing Xu'
Huawei Noah's Ark Lab, "Hong Kong University of Science and Technology Sun Yat-sen University
ABSTRACT
Unsupervised large-scale vision-language pre-training has shown promising ad- vances on various downstream tasks. Existing methods often model the cross- modal interaction either via the similarity of the global feature of each modality which misses sufficient information, or finer-grained interactions using cross/self- attention upon visual and textual tokens. However, cross/self-attention suffers from inferior efficiency in both training and inference. In this paper, we intro- duce a large-scale Fine-grained Interactive Language-Image Pre-training (FILIP) to achieve finer-level alignment through cross-modal late interaction mecha- nism, which uses a token-wise maximum similarity between visual and textual tokens to guide the contrastive objective. FILIP successfully leverages the finer- grained expressiveness between image patches and textual words by modifying only contrastive loss, while simultaneously gaining the ability pre-compute im- age and text representations offline at inference, keeping both large- sarge-scale training and inference efficient. Furthermore, we construct a new large-scale image-text pair dataset called FILIP300M for pre-training. Experiments show that FILIP achieves state-of-the -fetthrrrrt performance on multiple downstream vision-language tasks including zero-shot image classification and image-text retrieval. The visu- alization on word- patch alignment further shows that FII can learn meaningful fine-grained features with promising localization ability.
INTRODUCTION
Large-scale Vision-Language Pre-training (VLP) models like CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) have recently demonstrated success across various downstream tasks. They learn visual and textual representations from millions of image-text pairs collected from the Internet and show superior zero-shot ability and robustness. The core technique of these models lies in the global contrastive alignment of the images and texts through a dual-stream model. Such architecture inferencceefffcient for downstream tasks like retrieval because the encoders for the two modalities can be decoupled and the image or text representations can be pre-computed offline. However, CLIP and ALIGN model the cross modall interaction via solely the similarity of the global feature of each modality, lacking the ability of capturing finer-level information like the relationship between visual objects and textual words. In this paper, develop simple yet efficient cross-moda finer-grained interaction mechanism for large-scale VLP.
To achieve finer-grained cross-modal interaction, previous methods mainly exploited kinds of methods. (1) One line of work (Chen et al., 2020; Li et al., 2020b; Dong at., 2021; Li et al., 2021b; Zhang et al., 2021: Zhan al., 2021) uses pre-trained object detector to extract region-of-interee (ROI) features from images, and then fuses it with the paired text through VLP model. This design complicates the pre-training due to pre-computing and storing a large number of ROI features. In addition, the zero-shot ability of these approaches is usually limited by the predefined number of classes and their performance also restricted by the quality of the detector.
(2) Another line of work (Li et al., 202 Kim et al., 2021) enforces the token-wise or patch-wise representations from
"qqual contribution
Corresponding authors: xu.hang huawei.com, xdliang328 @annnn22@@@mmam @mmail.com

---

Published as a conference paper at ICLR 2022
both modalities into the same space and models these finer-grained interactions via cross-attention (Li et al., 2021a) or self-attention (Kim et al., 2021). However, these methods are usually less ef- ficient in terms of both training and inference. In particular, during training, cross-attention in (Li et al., 2021a) requires to be performed in an encoder-decoder structure, while the complexity of the self-attention (Kim et al., 2021) grows quadratically with the length of the prolonged concatenated sequences of both modalities. During inference, the data from both modalities are intertwined to compute the cross-attention or self-attention, and can not be pre-computed offline as dual-stream models like CLIP and ALIGN. This can be less efficient for downstream tasks like image/text re- trieval and image classification.
In this paper, we propose a large-scale Fine- tineggraind Interactive Language- Imagee---meeeeeee Pre-training frame- work named FILIPP to address these limitations. Inspired by Khattab & Zaharia (2020), we model the fine-grained semantic alignment through novel cross-modal late interaction mechanism in the contrastive loss, instead of using cross or self-attention. Specifically, our fine-grained contrastive learning uses a token-wise maximum similarity between visual and textual tokens to guide the con- trastive objective. In this way, FILIP successfully leverages the finer-grained expressiveness among image patches and textual words while simultaneously gaining the ability to pre-compute image and text representations offline. Unlike Khattab & Zaharia (2020), we discard the padded tokens and use average instead summation of token-wise maximum similarities when computing the image-text alignment, which enhances the cross-modal representation learning and stabilizes training. Further- more, we construct large-scale pre-training dataset named FILIP300M from the Internet. Data cleaning and image-text data augmentation are also explored and proved useful in this work.
Extensive experiments show that by effectively learning fine-grained representations, FILIP achieves stateoof-the-ar performance on multiple downstream tasks, including zero-shot image classification and image-text retrieval. For example, FILIP reaches 77.1% top-1 accuracy for cero-shot ImageNet classification, surpassing CLIP with less training data. Visualizations on word word-patch alignment further show that FII.IP learns meaningful finer-graine features with promising localization ability.
2 RELATED WORK
Langua-L Pre-training Models. The pre-train-and-fin-- fine-tune scheme has achieved great success in the domains of natural language processing (Devlin et al., 2019; Brown et al., 2020) and computer vision (Dosovitskiy et al., 2020). It is then naturally extended to a joint cross-modal do- Pre-training (VLP). The pre-training datasets of recent VLP models
main of include publically available datasets like YFCC100M (Thomee et al., 2016) and CC12M (Chang- pinyo et al., 2021). as well as larger-scale datasets with more than 100M samples in CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021), which are shown to be even more powerful. The pre- training tasks of VLP models can be categorized into two categories: image-text contrastive learning task and Language Modeling (LM) based tasks: (i) CLIP (Radford et al., 2021), ALIGN (Jia et al., 2021) and UNIMO (Li et al., 2021b) make use of cross-modal contrastive learning which aligns the textual and visual information into a unified semantic space; (ii) VisualBERT (Li et al., 2019), UNITER (Chen al., 2020), M6 (Lin et al., 2021), and DALLE- (Ramesh et al., 2021) employ LM-like objectives, including both masked LM (e.g., Masked Language/Region Modeling), and au- toregressive LM (e.g., image captioning, text-grounded image generation). On the other hand, some methods rely on pre-trained object detection model such as Faster-RCNN (Ren et al., 2015) to extract image regional features offline, which requires extra labeled bounding- moxding-ooxxx data and makes the approach less scalable. Recent efforts such as SOHO (Huang et al., 2021) and SLMVLM (Wang et al., 2021) try to eliminate this burden via visual dictionary or PrefixLM (Raffel et al., 2020). In this paper, we directly learn fine-grained vision-language representations in an end-to-end and simpler manner while maintaining the benefit of inference efficiency.
Multi-Modality Interaction Mechanism. The core of vision-language pre-training models lies in modeling the interaction between the two modalities. There are mainly two types of cross-modal interaction architectures: Single-stream models like VisualBERT (Li et al., 2019) and ViLT (Kim et al., 2021) directly concatenate the patch-wise regional visual features and textual embeddings and feed them to the transformer based model. Dual-stream models such as ViLBERT (Lu et al., 2019) and CLIP (Radford et al., 2021) have separate encoders for different modalities. This allows flexible use of different models for different modalities, and efficient inference for downstream tasks like tmage-text retrieval, through the ability of decoupling the encoders and pre-compute image/text
2

---

Published as a conference paper at ICLR 2022
Cross-modal Late Interaction
Image-to-text Contrastive
Text-to-lmage Contrastive
1. Max
Token-wise Similarity
Ti ..
7
12
Visual Token
Kig
xeye
Image Text-to-image
Textual Token
Positional Embedding
Image Encoder
Text Encoder
Linear Projection
Token Embedding
<BOS>
in
the basin <EOS>
Figure Overall architecture of FILIP, a dual-stream model with Transformer-baser image and text encoders. On top of the image and text encoders, the representations of textual tokens and visual tokens are linearly projected the multi-modal joint space. A novel fine-gratd fineggainedd contrastive learning equipped with cross-modal late interaction is proposed, which uses a token-wise maximum similarity between visual and textual tokens.
features offline. SCAN (Lee et al., 2018) considers latent alignments between image regions and words. However, it based on Triplet loss with a bottom-Up attention via Faster-RCNN to extract object features while we try directly learn to localize fine-grained object from patches. In this paper, while following the dual-stream approach for its flexible and efficient inference, we further propose new multi-modal interaction mechanism to capture the fine-graines representations.
METHOD
In this paper, we propose a new cross-modal pre-training model that excels in fine fine-grained interac- tion between image encoder and text encoder for mining more detailed semantic alignment, named as FILIP, as shown in Figure Particularly, FILIPP a dual-stream model with Transformer-based image and text encoders. For the visual modality, the image encoder is a Vision Transformer (Doso- vitskiy et al., 2020) which takes the concatenation an extra [CLS] token embedding and linearly projected image patches input. For the textual modality, following Radford et al. (2021), the lower-cased byte pair encoding (BPE) (Sennrich et al., 2016b) with a vocabulary size of 49,408 tokenize the text. Each text sequence starts with [BOS] token and ends with [EOS] token. After the word embedding layer, the token embeddings are fed into a modified decoder-only Transformer model as (Radford al., 2019). On top of the image and textual tokens and visual tokens are linearly projected to the multi-modal common space, and are separately L2-normalized. Different from existing dual-stream models (c.g.. CLIP and ALIGN) which models cross-modal interaction via only the global features of the entire image and text se- quence, we introduce novel fine fine-grained contrastive learning objective equipped with cross-modal late interaction which takes into account the fine-grained interaction between image patches and textual tokens, detailed in Section 3.1.
encoders, the representations of
3.1 FINE-GRAINED CONTRASTIVE LEARNING
Contrastive representation learning has recently been found to learn better representations than its predictive counterpart in both visual (Tian et al,, 2020) and vision- (Radford et al., 2021). Under a general formulation of cross-modal contrastive learning (Radford al., 2021), want lear encoders fo for image data I and ga for text data T such that, given an image x' I. and a text TT E T. the encoded representations fo(x') and (xx")) are close they are related and far apart not, under a distance metric. In each training batch, we sample image-text pairs (#/ a T 1 For image a. in image-text pair (#r az ag is its positive, while
moss-modal pre-training

---

Published as a conference paper at ICLR 2022
the other texts will be used in-batch negatives. The image-to-text contrastive loss C for a/ can then be formulated as
1 exp($k,,)
C (a (a)))==)))
- log Σ, exp(sk)))
where $k. denotes the similarity of the k-th image to the j-th text. Similarly, the text-to-imaae contrastive loss for 2 is
exp(sk
[ (a/))=11) - log E, exp(s)
The total loss of this mini-batch can be represented by
(4t + c.) L = 2 k=1
Σ
(1)
3.1.1 CROSS-MODAL LATE INTERACTION
From the contrastive loss (1), the modsl interaction is reflected in how we compute the sim- ilarities sf and sy, for the i-th image and j-th text. Previous methods like CLIP (Radford et al., 2021) and ALIIN (Jia et al., 2021) simply encode each image text separately to a global feature i.., fo(a:) ) E TR and go (x) E RRd, and compute these two similarities as
= = fo(x)) f ='(xa)""o0(
neglecting tner-grained interactions (e.g., word-patch alignment) between the two modalities. To alleviate this problem, while simultaneously maintain the training and inference efficiency dual- stream models, we apply cross-modal late interaction inspired by Khattab & Zaharia (2020) to model the token-wise cross modal interaction.
Specifically, denote nı and n2 the number of (non-padded) tokens of the i-th image and j-th text, respectively, and the corresponding encoded features are (=()) E I xd and go (xx)) E R For the k-th visual token, we compute its similarities with all textual tokens of 2j,, and use the largest one
max [fo Dl [9)(x)))r 0gr<ma
as its token-wis. fikee-wwieeee maximum similarity with 2T. We then use the average token-wise maximum similarity of all non non-padded tokens in the image (resp. text) as the similarity of an image to a text (resp. a text to an image). The similarity the i-th image to the j-th text can thus formulated as:
1 Σ Uo(a (ga(a)))) (m))
(4)
m k=
where mf arg maxosren (fo (x Me (90(0))) Similarly, the similarity of the j-th to the i-th image is
$ (0/ , m3) 12
Σ (a Ming (=o(=;)))
(5)
(x!,a))) an)) in Equation (4) does not where m = arg maxogrem ff((x)) /) OF Io (zz"))>= kk- Note that sf ( necessarily equal of, (a// az) in Equation (5).
Remark Intuitively, the token-wise maximum similarity in Equation (3) means that for each image patch, we find most similar textual token. Similarly, for each textual token, we also find closest image patch. By applying this to the similarity calculation in (4) and (5) for contrastive loss (1), the dual-stream model learns fine- grained alignment between image patches and textual tokens.
The original late interaction mechanism in (Khattab & Zaharia, 2020) computes the relevance score of token-wise maximum similari-
of document to query padded with mask tokens, ties, and is optimized via a pairwise softmax cross-entropy loss. Though inspired from Khattab & Zaharia (2020), our proposed cross-modal late interaction differs in several aspects. Firstly,
4

---

Published as a conference paper at ICLR 2022
exclude the padded textual tokens when computing the similarity, they harm the performance. We speculate that this is because these padded tokens also learn textual representations and will mislead the model to align image patches to these meaningless padded tokens rather than mean- ingful non-padded words. Secondly, when computing similarities (4) and (5), we use the average of the token-wise maximum similarities instead of summation in (Khattab & Zaharia, 2020). This is because the number of non-padded tokens varies from text to text, and this summation over all non-padded tokens can have quite different magnitudes, leading to less stabilized training and worse final performance. These two modifications are crucial to not only the downstream tasks' perfor- mance, but also the quality of the word word-patch alignment. more detailed discussion can be found in Appendix A.7. Thirdly, we optimize the late interaction mechanism via a contrastive loss (1) which found powerful vision-language pre-training (Radford al., 2021) instead of the original pairwise loss in (Khattab & Zaharia, 2020).
Training Efficiency. Though the cross-modal late interaction is able to capture finer-grained fea- tures compared with the original loss, it relies on the token-wise representations of both modalities, and can be inefficient in terms of communication, memory and computation, especially when the batch size is large. To alleviate this problem, we utilize several methods. Firstly, we reduce the embedding size to 256. Besides, we reduce the precision of the last-layer features of both modalities from fp32 to fp16 before node communication in distributed learning setting, and perform the mul- tiplication in Equations (4) and (5) under the reduced precision. In addition, since the complexity of similarity calculation scales with the sequence length of textual tokens and image patches, for each image (resp. text), we select the 25% tokens with the highest token-wise maximum similarity score (Equation (3)) among all texts (resp. images) in the same local worker before node communica- tion, based on the intuition that each sample can be represented by a few of the most representative tokens. Effects of these modifications are studied in Section 4.4.
3.1.2 PROMPT ENSEMBLE AND TEMPLATES
Due to the problem of polysemy and inconsistency with the pre-training process, following Radford al. (2021), we also use prompt templates to augment the original label for some downstream tasks. For visualizations, for simplicity, we use only one prompt template across the paper, i.e. "a photo of a [label)." as Radford et al. (2021). For other experiments, we report results using prompt ensemble following Radford et al. (2021). When multiple prompts are allowed, the token-wise representations of different prompt templates for the same class label are different, and can not be summed together to form a mean textual representation as in (Radford et al., 2021). Thus, instead of ensembling different prompt templates by their mean textual representation, we ensemble them by their mean token- wise similarity, Specifically, suppose there are C' prompt templates, each label is augmented to C different texts x xx. .... zzC. The similarity between an image x' and this label is computed
as + >C (x",,, x²"), where s!. is defined in Equation (4).
We use unified rule-based method inspired by Radford et al. (2018) to construct prompt templates for image classification tasks. Specifically, each template consists of four components:
[prefix] [label), [category description). [suffix].
(6)
Here, the "[prefix]" is an in-context description like "a photo of a" similar as Radford et al. (2021); "label" is a class label of the dataset; "[category description]" describes the category which is found helpful for some fine- greggrained image classification datasets (Radford et al., 2021), e.g.. " type of pet" for dataset Dxford-IIIT Pets. An interesting finding is that, adding suffix that includes the reference word "it" (c.g.. "I like it.") at the end of the prompt empirically improves the zero-shot classification performance of the proposed model. We speculate this is because the reference word "it" strengthens the fine-grained cross-modal alignment, as it can also be aligned to image patches of the target object. Detailed prompt templates for different datasets can be found in Appendix A.5.
3.2 IMAGE AND TEXT AUGMENTATION
To obtain better generalization and data-efficiency of the model, we perform data augmentation on both images and texts during the pre-training phase to construct more image-text pairs. We apply AutoAugment (Krizhevsky et al., 2012; Sato et al., 2015; Cubuk et al., 2019; Hoffer et al., 2020) for image augmentation, following the SOTA vision recognition methods (Touvron et al., 2021; Xie et al., 2020b). To ensure the augmented texts are semantically similar as the original one, for text
5

---

Published as a conference paper at ICLR 2022
Table I: Top-1 accuracy(%) of zero-shot image classification on 12 datasets. Our FILIP can boost 3~55 accuracy average.
StanfordCarss
CIFARIO
CIFAR100
Caltech101
Flowers102
Fond101
SUN399
Airerufts
OxfordPets
EuroSAT
ImageNet
Average
DTD
CIJIP-VIT-B/32
91.3
65.1
87.9
59.4
66.7
84.4
63.2
44.5
21.2
87.0
49.4
63.2
65.3
FILIP buse ViT-B/32
86,9
65.5
91.9
55.4
85.3
82.8
69.1
49.3
57.2
88.1
49.9
68.8
70,9
CLIP-ViT-L/4
96.2
77.9
92.6
77.3
78.7
92.9
67.7
55.3
36
93.5
59.9
75.3
75.3
FILIP hage VIT-L/14
95.7
75.3
93.0
70.8
90.1
92.2
73.1
60.7
60.2
92
59.2
77.1
78.3
augmentation, we rewrite the original text using back-translation (Xie et al., 2020a; Sennrich et al., 2016a). Specifically, the texts are first translated to the target language and then translated back to the source language. We choose German and Russian as the target language and get extra two texts for each image-text pair. When constructing batch of image-text pairs during the pre-training, the text of each image-text pair is randomly sampled from the three candidate texts, i.e., the original text and two bock-translated texts.
3.3 PRE-TRAINING DATASET
A sufficiently large image-text dataset is prerequisite for vision-language pre-training. Recent CLIP (Radford et al., 2021) and ALIGN (Jia et al., 2021) construct datasets with 400M and 1800M image-text pairs, respectively. In this work, also collect large-scale dataset called FILIP300M from the Internet, which consists of 300M image-text pairs and covers board vision and language concepts. For image-based filtering, we remove the images whose shorter dimension is smaller than 200 pixels and the aspect ratio is larger than 3. For text-based filtering, we keep only English texts, and exclude the meaningless ones, e.g., img-0.jpg. We also discard image-text pairs whose texts are repeated for over 10 times. Besides, we also use 3 public datasets, including Conceptual Captions 3M (CC3M) (Sharma et al., 2018), Conceptual 12M (CC12M) (Changpinyo et al., 2021) and Yahoo Flickr Creative Commons 100M (YFCC100M) (Thomee et al., 2016). We apply the same filtering rules on YFCC100 Finally, we use about 340M image-text pairs for pre-training. Despite using smaller training dataset than CLIP and ALIGN, our models still outperform them in most down-steam tasks (see Section 4).
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Model Architectures. We train two models from scratch, i.e., FILIP and FILIP turge. The model architectures follow CLIP (Radford et al., 2021), i.e., the image encoder is ViT-B/32 for FILIPoase and ViT-L/14 for FILIP large- More details can be found in Appendix A.3.
Pre-training Details. To save memory and scale up the batch size, automatic mixed-precisiie (Micikevicius et al., 2018) and gradient checkpoint (Griewank & Walther, 2000; Chen et al., 2016) are used The input images are resized to 224 x 224 resolution during pre-training and the maximum length of the text is limited to 77 tokens following Radford et al. (2021). The training is mainly conducted on Nvidia V100 GPUs and Ascend Cards. FILIP base is trained on 128 cards about days and FILIP large takes about 24 days train 192 cards. Unless otherwise specified, FILIP to compare with other methods and FILIP for ablation. We train both models using the LAMB optimizer (You et al., 2020) and cosine learning rate schedule (Loshchilov & Hutter, 2016) with a linear warmup. Weight decay regularization is applied to all parameters except bias, layer normalization, token embedding, positional embedding and temperature in contrastive loss. Detailed values of hyperparameters for different datasets and models can be found in Appendix A.3.
4.2 IMAGE CLASSIFICATION
In this section, we compare our FILIP with CLIP (Radford et al., 2021) on 12 downstream image classification datasets.

---

Published as a conference paper at ICLR 2022
Table 2: Results of zero-shot image-text retrieval on Flickr30K and MSCOCO datasets. The last two rows (marked with *) report the zero-shot results on Flickr30K dataset of model fine-tuned on MSCOCO dataset, following the setting of ALBEF (Li et al., 2021a).
Flickr30K
MSCOCO
image-to-text
text-to-image R@5
text-to-image R@l R@5 R@ R@1 R@5 R@ 10
image-to-text
R@1 1 R@5
R@ 10
R@I
R@ 10
Unicoder- VL
64.3
85.8
92.3
48.4
76.0
85.2
ImageBERT
70.7
90.2
94.0
54.3
79.6
87.5
44.0
71.2
80.4
32.3
59.0
70.2
UNITER
83.6
95.7
97.7
68.7
89.2
93.9
CLIP
88.0
98.7
99.4
68.7
90.6
95.2
58.4
81.5
88.1
37.8
62.4
72.2
ALIGN
88.6
98.7
99.7
75.7
93.8
96.8
58.6
83.0
89.7
45.6
69.8
78.6
FILIP
89.8
99.2
99.8
75.0
93.4
96.3
61.3
84.3
90.4
45.9
70.6
79.3
ALBEF
94.1
99.5
99.7
82.8
96.3
98.1
FILIP*
95.4
99.8
100.0
84.7
97.0
98.7
Zero-shot Classification. As in Section 3.1.2, we apply of prompts (Appendix A.5) for each dataset and ensemble them to get the final results. Table shows the results on 12 datasets. Despite using less training data (340M vs. 400M), both FILIP base and FILIP large considerably outperform their CLIP counterparts in terms of average top-1 accuracy over 12 datasets, i.e., achieving absolute improvements of 5.6% and 3.0%. respectively. In particular, our FILIP surpasses CLIP on ImageNet, the largest dataset among 12 datasets. FII.PP also achieves substantial performance gains on some domain-specific datasets like Aircrafts. We speculate this is because, unlike CLIP which aggregates the information of the whole image into the [CLS] token, our proposed FILIP focuses more on the target object by directly aligning the image patches the target object with the textual tokens corresponding to the class label (visualizations of word-patch alignment are in Section 4.5).
Linear Probe. Table 14 in Appendix A.6 shows the linear probe results, and FILIP again outper- forms CLIP by 1.2%118% points on average. More details can be found in Appendix A.6.
4.3 IMAGE-TEXT RETRIEVAL
Image-text retrieval consists of two sub-tasks: image-to-text retrieval and text-to-image retrieval. We evaluate our FILIP model on two retrieval benchmark datasets: Flickr30K (Plummer et al., 2015) and MSCOCO (Lin et al., 2014), under both zero-shot and fine-tuned settings. More details of experimental setting can be found in Appendix A.3.
Tables and 3 show the results of zero-shot and fine-tuned image-text retrieval, respectively. We compare our FILIP model against methods with complex attention layers including Unicoder-VL. (Li at., 2020a), ImageBERT (Qi et al., 2020), UNITER (Chen et al., 2020), VILLA (Gan et al., 2020), ERNIE-ViLe (Yu et al., 2021), Oscar eLi al., 2020b), VinVL (Zhang al., 2021), ALBEF (Li et al., 2021a), and methods trained on larger-scale image-text datasets including CLIP (Radford et al., 2021) and ALIGN (Tia al., 2021). As we can see, FILIP achieves state-of-the-art performances under all metrics on both Flickr30K and MSCOCO datasets, except for zero-shot text-to-image retrieval on Flickr30K, where FILIP achieves competitive performance with SOTA. For zero-shot image-to-text retrieval on MSCOCO dataset, the absolute R @@) I of our proposed FILIP is 2.7% higher than ALIGN, which is trained on much larger dataset.
4.4 ABLATION STUDY
Effectiveness of Each Component. We study the effectiveness of each component in FILIP, i.e., image/text augmentations and cross-modal late interaction. Experiments are conducted on FILIP with a filtered subset of YFCC100M as the training dataset (as described Section 3.3), on both zero-shot retrieval and classification tasks. We measure models' performance MSCOCO zero- shot image-text retrieval and ImageNet zero-shot classification, which are two effective indicators for the quality of the learned vision-language representations.
Table reports the results. As can be seen, all three components are beneficial for both tasks. Despite the simple design, cross-modal late interaction brings significant performance improvements over the baseline (the vanilla CLIP VIT-B/32), with an absolute R@ gain of 5.5% (resp. 3.8%) for image-to-text (resp. text-to-image) retrieval MSCOCO and absolute top-1 accuracy gain

---

Published as a conference paper at ICLR 2022
Table 3: Results fine-tuned image-text retrieval on Flickr30K and MSCOCO datasets.
Flickr30K
MSCOCO
image-to-text R@5
text-to-image R@5
image-to-text R@5
text-to-image
R @
R@ 10
R@1
R@ 10
R@1
R@ 10
R@1
R@5
R@
10
Unicoder- VL
86.2
99.0
71.5
90.9
94.9
62.3
87.1
92.8
76.7
85.9
ImageBERT
87.0
97.6
99.2
73.1
92.6
96.0
66,4
89.8
94.4
50.5
78.7
87.1
UNITER
87.3
98.0
99.2
75.6
94.1
96.8
65.7
88.6
93.8
52.9
79.9
88.0
VILLA
87.9
97.5
98.8
76.3
94.2
96.8
ERNIE- ViL
88.1
98.0
99.2
76.7
93.6
96.4
Oscar
73.5
92.2
96.0
57.5
82.8
89.8
VinVL
75.4
92.9
96.2
58.8
83.5
90.3
ALIGN
95.3
99.8
100.0
84.9
97.4
98.6
77.0
93.5
96.9
59.9
83.3
89.8
ALBEF
100.0
85.6
97.5
98.9
77.6
94.3
97.2
84.3
90.5
Our FILIP
96.6 100.0
100.0
87.1
97.7
99.1
78.9
94.4
97.4
61.2
84.3
90.6
Table Ablation study of different components on pre-training subset of YFCC100M. 12T and T2I are abbreviations for image-to-tex, and text-to-image retrieval, respectively. "ZS" means zero-shot performance. Underlined numbers have the highest improvements for the corresponding metries.
MSCOCO
ImageNet ZS Top1
Model
12T R@1
12T R@S
T21 R @@
T2I
R@5
Baseline (VIT-B/32)
25.0
49.5
14.7
34.7
30.4
w/ image augmentation
26.1
51.8
16.5
37.5
32.5
w/ back translation
29.2
55.0
17.9
39.8
33.9
w/ cross-modal late interaction
30.5
55.3
18.5
40.0
34.3
Our FILIP base
33.4
60.1
23.0
46.2
37.8
Table 5: Efficiency study of the cross mosdll late interaction. "orig" and "late" stand for the con- trastive loss based the original cosine similarity in CLIP and our proposed cross-modal late interaction, respectively. "ZS" means zero-shot performance. We report results for VIT-B/32 trained on filtered YFCC100M with V100 GPUs, with batch size 512 per GPU. Training time and memory consumption are tested using the same gradient checkpoint configuration. denotes our
final setting used in other experiments.
Embed dim
Embed precision
Token
Training time (sec/iter)
Memory (MB)
ImageNet ZS Top1
Loss
%
orig (baseline)
512
fp32
1.31
14300
30.4
late
512
fp32
100%
2.85
26000
34.6
late
512
fp16
100%
2.67
23468
34.5
late
256
fp16
100%
2.31
22382
35.2
late
256
fp16
50%
1.61
16336
34.5
late*
256
fp16
25%
1.39
16100
34.3
of 3.9% for zero-shot classification ImageNet. Further improvements are observed when all components are combined together.
Efficiency Study of Cross-modal Late Interaction. Since the late interaction mechanism in Sec- tion 3.1.1 requires to calculate the similarity between all visual and textual tokens, its efficiency can be a problem when employed in large-scale distributed training. As described Section 3.1.1, make several attempts to address the issue. Table 5 shows the efficiency improvement on zero-shot classification on ImageNet when these attempts are applied. As can be seen, these attempts improve the efficiency of late interaction without accuracy drop. Combining all three attempts achieves only slightly slower training and larger memory consumption than the original loss CLIP.
4.5 VISUALIZATION OF FINE-GRAINED ALIGNMENT
In this section, we visualize FILIP's capability of capturing fine-grained cross-modal correspon- dence using the method of word- patch aligmment. To make fair comparison, trained on YFCC100M and CLIP's ViT-B/32, which are the same size, for visualization. Each image is patchified 7x7 image patches. More visualization results can be found in Appendix A.4.
FILIP
8

---

Published as a conference paper at ICLR 2022
Raw image
FILIP
CLIP
Raw image
FILIP
CLIP
(a] Balloon (5)
(b) Lifeboat (5)
Raw image
FILIP
CLIP
Raw image
FILIP
CLIP
(c) Small white butterfly (5, 6, 7)
(d) Electric locomotive (5,6)
Figure 2: Visualizations of word-patch alignment for classes of the ImageNet dataset and "a photo of {label)." is the prompt. Numbers in the parentheses after the class label indicate the location indices of the class label in the tokenized textual sequence. The correct predictions are highlighted by opaque patches with the class label indices in red.
Visualization Method. The word-patch alignment is performed based on the token-wise similarity between the image patches and textual tokens. Specifically, for the k-th image patch, the location index of textual token with the largest similarity with it (m in Equation (4)) is considered as its example. There balloon. [EOS]", and the location
predicted label, and is placed the center of it. Take class "balloon" 8 tokens in the tokenized textual sequence "[BOS] a photo index of the class label "balloon" is "5". Note that one class label may be tokenized to more than one token. Location indices of textual tokens corresponding to the class label are highlighted in red, while the others are marked white. desired model that learns fine- seee-graindd representations would predict image patches of the target object to red indices.
Figure shows the word- patch alignment results for FILIP and CLIP on classes from the ImageNet dataset. As can be seen, FILIP exhibits the finer-grained understanding of an image in the following aspects. (i) A single object: From the visualization of class "small white but- terfly", the image patches covering the object are all classified correctly; (ii) Same object in different shapes: From the visualizations of class "balloon" and "lifeboat", image patches corresponding to all target objects with different shapes and locations are correctly classified; (iii) Key Components of an object: For class "electric locomotive", there are two key components crucial to correctly classifying the image, i.e., "electric" and "locomotive", whose corresponding textual token indices are "5" and "6", respectively. As can be seen, image patches matching these two key components respectively correctly classified. On the other hand, CLIP can not correctly align image patches with corresponding textual tokens. Compared with Kim et al. (2021) which uses an extra optimal transport to align the textual word and image patch distributions, the word-patch alignment can be simply automatically learned by our method.
Observations.
CONCLUSION AND FUTURE WORK
This paper introduces FILIP, a simple yet generic framework towards fine-grained vision-language pre-training. By using a token-wise maximum similarity, our method learns fine-graines represen- tation for patches in the images and words in the sentences. While it achieves competitive results against several large-scale multi-modal pre-training on various downstream tasks, both its archi- tecture and training procedure can still be optimized to improve its performance. In the future, more advanced image encoder as well as a well-designed interaction layer can be used boost the performance. Furthermore, generation tasks. To this end, we hope to extend FILIP as generic and unified interface for solving large variety of languae tasks.
further add more masked language/image loss to support more

---

Published as a conference paper at ICLR 2022
REFERENCES
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in neural information processing systems, 2020.
Soravit Changpinyo, Piyush Sharma. Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web- scale image-text pre-training to recognize long-tail visual concepts. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost. preprint urXi:1660.061111 2016.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference computer vision, pp. 104-120. Springer, 2020.
Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 113-123, 2019.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Annual Conference of the North Amer- ican Chapter of the Association for Computational Linguistics, 2019.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering ext-to-image generation via transformers. arXiv preprint 2rXiv:2105.132000 2021.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp- 1422-1430, 2015.
Xiao Dong, Xunlin Zhan, Yangxin Wu, Yunchao Wei, Xiaoyong Wei, Minlong Lu, and Xiaodan Liang. MSproduct: A multi-modal pretraining benchmark for e-commercial product downstream tasks. Preprint arXiv:2109.04275, 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Representations, 2020.
Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for -nd--annnddgggeeee representation learning. arXiv preprint arXiv:2006.061 1955,, 2020.
Andreas Griewank and Andrea Walther. Algorithm 799: revolve: an implementation of check-
pointing for the reverse adjoint mode of computational differentiation. ACM Transactions Mathematical Software (TOMS), 26( (961))194440 45, 2000.
Elad Hoffer, Tal Ben-Nun, Itay Hubara, Niv Giladi, Torsten Hoefler, and Daniel Soudry. Aug-
ment your batch: Improving generalization through instance repetition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8129-8133 38, 2020.
Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision- -aiiuag representation learning. IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12976-1 12985, 2021.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Chao Jia, Yinfei Yang. Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc V Le, Yunhsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and lision-language representation learning with noisy text supervision. In International Conference Machine Learning, 2021.
10

---

Published as a conference paper at ICLR 2022
Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 39-48, 2020.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and ssiionand-langguage transformer without convo-
lution or region supervision. In International Conference on Machine Learning, 2021.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, volume 25, pp. 1097-1105, 2012.
Kuang- Hueng Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 201-216, 2018.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang. Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 11336-11344, 2020a.
Junnan Li, Ramprasaath R. Selvaraju, Akhilesh Deepak Gotmare, Shafiq Joty, and Steven Xiong, Caiming andHoi. Align before fuse: Vision and language representation learning with momentum distillation. Technical Report ar Xiv:2207.007551 2021a.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kei- Chang. Visualbert: A simple and performant baseline for vision and language. Preprint arvi: I 998.03557, 2019.
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, and Haifeng Wang- Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learn- ing. In Annual Meeting of the Association for Computational Linguistics, 2021b.
Xiujun Li, Xi Yin, Chumyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Osear: Object-semantics aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pp. 121-137. Springer, 2020b.
Junyang Lin, Rui Men, An Yang, Chang Zhou, Ming Ding, Yichang Zhang, Peng Wang, Ang Wang, Le Jiang, Xianyan Jia, et al. M6: chinese multimodal pretrainer. Preprint arXiv:2103.00823, 2021.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
I. Loshchilov and F. Hutter. Sgdr: Stochastic gradient descent with warm restarts. ICLR 2017 (Sth International Conference on Learning Representations) 2016.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: pretraining task-agnostic visiolin- guistic representations for lisionaand-language tasks. In International Conference on Neural Information Processing Systems, pp. 13-23, 2019.
Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg. Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, al. Mixed precision training. In International Conference on Learning Representations, 2018.
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathieu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, et al. Seikit-learm: Machine learning in python. the Journal of machine Learning research, 12:2825-2830. 2011.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svet- lana Lazebnik. Flickr30k entities: Collecting region-to- to-phrase correspondences for richer image- to-sentence models. In IEEE international conference computer vision, pp. 2641-2649, 2015.
Di Qi, Lin Su, Jia Song, Edward Cui, Taroon Bharti, and Arun Sacheti. Imagebert: Cross-modal pre-training with large-scale weak weak-supervised image-text data. arXiv preprint arXiv:2001.07966, 2020.
11

---

Published as a conference paper at ICLR 2022
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language under- standing by generative pre-training 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, llya Sutskever, et al. Language models are unsupervised multitask learners. OpenAl blog. 1(8):9, 2019.
Alec Radford, Jong Wook Kim, Chris Hallaey, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Preprint MrXiv:2103.00020 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67, 2020.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. Preprint arXiv:2102. 12092, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28: 91-99, 2015.
Ikuro Sato, Hiroki Nishimura, and Kensuke Yokoi. Apac: Augmented pattern classification with neural networks. Technical Report arXiv:1505 0000020..222.... 2015.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In IEEE international conference on computer vision, pp. 118-626, 2017.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Improving neural machine translation models with monolingual data. In Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 86 86-96, 2016a.
Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1715-1722 2016b.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypemymed, image alt-text dataset for automatic image captioning. In Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556-255555 2565, 2018.
Bart Thomee, David Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. Yfec100m: The new data in multimedia research. Communications of the ACM, 59(2):64-73, 2016.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. In Computer Vision-ECCV 2020: 16th European Conference, pp. 776-794. Springer, 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Trriining data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. Preprint arXiv:2108. 10904, 2021.
Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised data augmentation for consistency training. In Advances in Neural Information Processing Systems, volume 33, 2020a.
Qizhe Xie, Minh-Thang Thang Luong, Eduard Hovy, and Quoc V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10687-10698, 2020b.
12

---

Published as a conference paper at ICLR 2022
Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. Large batch optimization for deep learning: Training bert in 76 minutes. In International Conference Learning Representattons, 2020.
Fei Yu, Jiji Tang, Weichong Yin, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. Ernie-vi:: Knowl- edge enhanced languae representations through scene graphs. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. -208-3216, 2021.
Xunlin Zhan, Yangxin Wu, Xiao Dong, Yunchao Wei, Minlong Lu, Yichi Zhang, Hang Xu, and Xiaodan Liang. m:: Towards weakly supervised instance-level product retrieval via cross- modal pretraining. In International Conference on Computer Vision, 2021.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang. Yejin Choi, and Jianfeng Gao. VinvI: Revisiting visual representations in vision-language models. In IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5579-5588, 2021.
13

---

Published as a conference paper at ICLR 2022
A APPENDIX
A. ETHICAL ISSUES IN DATA COLLECTION
When collecting the large-scale image-text pairs from the Internet, we perform person-name substi- tutions as in Changpinyo et al. (2021), in order to protect the privacy of the individuals appearing in the text. Specifically, we replace each person name appeared in the text with special < person > token. Besides, we also disgard image-text pairs whose text contains sensitive words.
A.2 DATASETS SUMMARY
Table 6 shows the number of image-text pairs of each datasets used in different pre-training methods.
Table 6: Number of image-text pairs used in the pre-training of FILIP, CLIP and ALIGN.
FILIP
CLIP
ALIGN (Jia al., 2021)
CC3M
CC12M
YFCCI00
FILIP300M 300M
(Radford et al., 2021)
#
3M
1OM
26M
400M
1800M
A.3 DETAILED EXPERIMENTAL SETTINGS
Table 7: The architecture parameters for FILIP models.
Embedding dimension
Input resolution
Image Encoder
Text Encoder
Model
#layers
width
#heads
#layers
width
#heads
FILIP Posse
256
224 x 224
12
768
12
12
512
8
FILIP Large large
256
224 224
24
1024
16
12
768
Model Architectures. We follow the same architecture design as CLIP, for both FILIP and FILIP large. except that reduce the embedding dimension from 512/768 256 for the efficiency of loss computation. Table 7 describes the details of architectures.
Details for Pre-training and Hyperparameters. For the implementation of the contrastive loss, following CLIP (Radford et al., 2021) and ALIGN (Jia al., 2021), we also set the temperature in the softmax function to be a learnable parameter and initialize it as 0.07. For the pre-training, the LAMB optimizer implemented by the cybertronai's 's open-source repository (https: //github. com/ com/cybeere cyberttronnna /ptppytoee orytorch-ai lamb). For the learning rate scheduler, we first assign base learning rate and then linearly warm it up the peak learning rate according the
We note that large weight decay is crucial to stabilize training and improve generalization. Specifically, we found that the training stability is challenging issue when applying mix precision training large-scale models, i.c., the training is extremely unstable and the NaN loss easily happens. Recent works
effective total batch size by square root strategy, peak 1r base x
botal.bs 512
Table 8: Common hyperparameters used for FILIP pre-training.
Hyperparameter
Value
Vocabulary size
49408
Initial temperature
0.07
LAMB 31
0.9
LAMB B2
0.999
LAMB
10-4
Warm-up iters
3000
Training epochs
30
14

---

Published as a conference paper at ICLR 2022
Table 9: Model- and dataset-specific hyperparameters used for FILIP pre-training. Numbers in batch size represent the total batch size across all workers and are calculated as: batch size per GPU x #GPUs. FILIP340M is the combination of FILIP30MM, YFCC CC12M and CC3M.
Model
Dataset
Batch size
Base LR
Weight decay
FILIP base
YFCC100M
1024 x 8
6 X 10-3
3e-2
FILIP base
FILIP340M
320 x 128
2 x 10-3
3e-3
FILIP large
FILIP340M
160 x 192 1.5 10
3e-3
DALL-E (Ramesh et al., 2021) 1) and Cogview (Ding ct al., 2021) also notice this issue and provide their solutions. However, found that simply increasing the weight decay and applying the trick of removing the weight decay of specific parameters as described in Section 4.1 work for our case. The base learning rate and weight decay are selected manually via observing the performance the early training stage. Table summarizes the common hyperparameters and Table 9 shows the model- and sppasct-specific hyperparameters for FILIP pre-treee pre-training.
Details for Image-text Retrieval. Following previous works (Jia et al., 2021; Li et al., 2021a), for Flickr30K, we test on the IK test set with or without fine-tuning on the 30K training set, while for MSCOCO, we test on the 5K test set with without fine-tuning on the 113K training set. We use the similarity between image and text for ranking and use the contrastive loss for fine-tuning. Since there multiple texts for each image in these two datasets, we change the ground-truth label of contrastive loss to consider multiple positives, by assigning probability of 1/#positive to each positive following ALBEF (Li al., 2021a). Besides, we also prompts during evaluation for both datasets, see Appendix A.5 for details. Table 10 shows the hyperparameters for image-text retrieval fine-tuning.
Table 10: Hyperparameters used for image-text retrieval fine-tuning.
Hyperparameter
Value
Image size
392 x 392
Training epochs
3
Optimizer Batch size
LAMB
5120
Base LR
10
Weight decay
3 x 10
A.4 MORE VISUALIZATIONS OF WORD- PORDH ALIGNMENT AND GRAD-CAM HEATMAPS
In Figure 3, we visualize the cross-modal alignment of the proposed method for more images, in terms of both word-patch aligmment as described Section 4.5 and Grad-CAM heatmaps (Selvaraju et al., 2017). We compute the Grad-CAM heatmaps based on the average self-attention maps the image patches classified to targeted textual tokens (i.c., the textual token(s) corresponding to the class label in the ImageNet dataset) in the last layer of the image encoder. We average the heatmaps over all attention heads. As can be seen, our proposed model learns meaningful alignment between image patches and textual tokens.
A.S PROMPT TEMPLATES FOR DOWNSTREAM TASKS
Image Classification. Table 11 shows the prompt templates for different image classification datasets in the form of " [prefix] [label), [category description]. [suffix]. " in Equation (6). There are three components to be determined in the template, i.e., the prefix, the category description and the suffix. For each component, we select several well-performed ones for each dataset. Then we use the full combinations of all three components the set of prompt templates for ensemble. For instance, we use 5 prefixes, no category descriptions, and 6 suffixes for dataset ImageNet. Then the total number of prompt templates for this dataset is: 5 6 = 30.
15

---

Published as a conference paper at ICLR 2022
Raw image
Patch prediction
GradCAM
Raw image
Patch prediction
GradCAM
Bald eagle (5,6)
Killer whale
0
0
(5,6)
Bullock cart (5,6)
Damselfly (5.6,7)
Firebost (5,6)
Necklace (5)
Mosque (5)
20
D
Yellow lady's slipper (5.6,7,8)
D
O
Q
Figure 3: More visualizations on different classes of ImageNet dataset. Numbers in the parentheses after the class label indicate the location indices of class label in the tokenized textual sequence.
16

---

Published as a conference paper at ICLR 2022
Table 11: Prompt templates used for 12 downstream image classification tasks.
Category description
Datasct
Prefix
Suffix
"a photo of "a jpeg photo of a", "a painting of a", "itap of a". "graffiti of a", "a cartoon", "a doodle"
None, "It's common in daily life", "It's cutc", "It's ugly", "It's weird", "Hope you like it"
CIFARIO
None
"a jpeg photo of a". painting of a", "a good photo of a", "a bad photo of a", "a photo of a". "itap a", "a rendering of a"
None, "It's common daily life", "It's bcautiful", "It's ugly". "I like it", "I take it to- day"
CIFAR100
None
"a photo photo of a", "a good photo of a". "a bad photo af
a", "a cropped
None, like it", "I hate it", "It's ugly", "It's cute"
Caltech101
None
like it", "It belongs my friend". "It's brand new", "It's popular recently", "It's impor- tant to me", take it today"
"a photo of a", "a closc-up photo of a", good photo of a", "a bad photo a"
"a type of car", type of auto- mobile"
Stanford- Car
"a photo (many) ". ren- dering of a (many) "itap of a (many)
type of flower", "a type of bloom"
"It's beautiful", "It's from my best friend", "It gives out a sweet perfume/frrarracee"
Flowers 102
"I like it", "It's common in daily life", "It's not common in daily life", "It's ugly", "It's cute", "It's beautiful"
"a photo of a", "a good photo of a", "a bad photo of a", close- up photo a", "itap of
ImageNet
None
made it today", like it", hate it", "It's delicious", "It's with nice flavour". "It's with terrible flavour", "It's popular recently"
"a type of food". "a type of nourish- ment"
"a photo of my", "a close-up photo of my", "itap of my"
Food101
"a pa of a", "a good photo of a", "a bad photo of a", "a bright photo a", a dark photo a", "a black and white photo of a". "a nice scene of a", "a terrible
None, like it", "I hate it", "It's beautiful", "It's common in daily life", "It's important to me"
SUN397
None
scene
"texture",
None, "It's out of style", "It's popular in old days", "It's ugly", "It's beautiful"
"itap of a", "a close-up photo of a"
DTD
"surface" "material"
type of plane", "a type of aircraft". "a type airliner"
"a photo of the", "a close-up photo of the". "a good photo of the ", "a pixelatee photo of the"
None," like it", "It's important me". "I take today", "Hope you like it"
Aircrafts
"a photo of my", "a reso- lution photo of my", "a good photo of my"
type of pet", "a type of dog cat"
None, "It's cute". "It's impor- tant to me", "I like it", "It's beautiful"
Oxford Pet
"a photo ofa", painting af
None, "an ample of aerial satellite im- ages"
None, "I like it" "It's taken from an aircraft or some flying object", "It's collected by imag- ing satellites"
"a cropped photo a", "a good photo of a", "a blurry photo of
EuroSAT
17

---

Published as a conference paper at ICLR 2022
Table 12:
Prompt templates used for zero-shot image-text retrieval Flickr30K and MSCOCO
datasets.
Dataset
Task
Prefix
Suffix
image-to-text retrieval
"a good photo of the" "a good photo of""
"I hate it." None
Flickr30K
text-to-image retrieval
image-to-texi retrieval (ext-to-image retrieval
"a good photo of"" None
"It is ugly." None
MSCOCO
Image-text Retrieval. Following CLIP (Radford et al., 2021), we use prompt in zero-shot image- text retrieval for both Flickr30K and MSCOCO datasets. The prompt is selected by the same rule as described in Section 3.1.2, except that we do not use "[category description]" here. Table 12 shows the prompt templates for zero-shot image-text retrieval on Flickr30K and MSCOCO datasets.
A.6 LINEAR PROBE ON IMAGE CLASSIFICATION
In this section, evaluate FILIP the linear probe for image classification. Following common linear probe setting, we freeze the whole backbone network and only finetune the last linear classi- fier. Since we remove the "[CLS]" token in our vision encoder, apply a mean pooling over all the other visual tokens to aggregate them into global image representation which is then fed into the linear classifier.
Setting. Following CLIP, we train the logistic regression classifier using scikit-learn's 's L-BFGS implementation (Pedregosa al., 2011), 1), with maximum 1,000 iterations on those 11 datasets except ImageNet. For ImageNet, we use pytorch-based codehase to accelerate the training with GPU. Following Doersch et al. (2015), we adopt Batch Normalization (loffe & Szegedy, 2015) layer before the linear classifier which is beneficial to stabilize the mixed-precisse resized crop and horizontal flipping are used to augment training data. We use the cosine learning rate scheduler with linear warmup of 10 epochs. More hyperparameters used in linear probe ImageNet are shown in Table 13.
training. Random
Table 13: Hyperparameters used for linear probe image classification on ImageNet.
Hyperparameter
Value
Image size
224
224
Training epochs
90
Optimizer Batch size
SGD
4096
Base LR
0.1
Weight decay
0
Results. Table 14 compares the linear probe performance of our proposed FILIP with CLIP over 12 datasets. Our FILIP base (resp. FILIP achieves 85.5% (resp. 91.0%) average Top-1 accuracy over downstream tasks, which provides noticeable improvements, i.e., 1.8% (resp. 1.2%) higher, compared to its CLIP's counterpart. This implies that our FILIPP learns more powerful vision features which may potentially facilitate border downstream vision tasks.
A.7 COMPARISON WITH KHATTAB & ZAHARIA (2020)
As is stated in Section 3.1, compared to Khattab & Zaharia (2020), besides being the first to ap- ply the late interaction to contrastive learning for vision-language pre-training, we make two other modifications, i.e., removing padded tokens and using average over non-padded tokens instead of summation. In the following, we show that these two modifications are crucial to the performance, and the quality of finer-granular word-patch alignment.
For comparison, we replace the proposed cross-mit cross-modal late interaction in FILIP base with the original late interaction in Khattab & Zaharia (2020). Following the setting in Section 4.4, pre-train
18

---

Published as a conference paper at ICLR 2022
Table 14: Top-1 accuracy(%)) of linear probe on image classification on 12 datasets. Our FILIP
outperforms CLIP by 1.2% points on average.
StanfordCars
CIFAR100
Caltech101
Flowers
OxfordPets
CIFARIO
Food101
SUN397
Aircrafts
EuroSAT
ImageNet
Average
DID
CLIP-ViT-B/33
95.1
80.5
93.0
81.8
96.6
88.8
76.6
76.5
52.0
90.0
97.0
76.1
83.7
FILIP base
95.3
80.3
95.0
78.6
98.7
86.2
77.9
78.1
76.6
88.0
95.9
75.8
85.5
CLIP-VIT-
V1T-L44
98.0
87.5
96.5
90.9
99.2
95.2
81.8
82.1
69.4
95.1
98.2
83.9
89.8
FILIPP lurge
97.9
87.0
97.2
89.0
99.6
94.6
83.2
83.9
84.8
93.5
97.3
84.5
91.0
the filtered YFCC100M with mixed-precision using 8 V100 GPUs. The batch size per GPU is 512 and the dimension of the token feature 256. We report results with the top 25% tokens (selected using the method in Section 3.1) during training. Note that the original late interaction in Khattab & Zaharia (2020) sensitive to the temperature in the softmax function, and we report the best result among several initialization values of the temperature.
Effect to Performance. Table 15 shows the comparison on zero-shot ImageNet classification. When these two modifications are removed, the zero-shot Top-1 accuracy of ImageNet drops from 34.3 to 32.7.
Table 15: Comparison of Top-1 Accuracy(%) between the proposed cross-modal late interaction loss and Khattab & Zaharia (2020) zero-shot ImageNet classification.
ours
late interaction Khattab & Zaharia (2020)
34.3
32.7
Effect to the Word-patch Alignment In Figure 4, compare the word-patch alignment using the models trained with the proposed cross-modal late interaction and the late interaction in Khattab & Zaharia (2020). According to the visualizations, using the original late interaction Khattab & Zaharia (2020) leads to less accurate word-patch alignment. Specifically, the object patches are often aligned to the padded tokens instead of class names. We speculate this because the padded tokens existing key textual tokens, similar to the finding in Section 3.2 of Khattab Zaharia (2020) that padding with masked tokens (which is called "query augmentation' in Khattab & Zaharia (2020)) tend to "re-weigh existing terms based on their importance for matching the query".
learn similar representations
A.8 ABLATION ON THE FULL PRE-TRAINING DATASET
In Table 16, we compare the proposed cross late interaction loss with the original CLIP loss (Radford et al., 2019) the full pre-training dataset introduced in Section 3.3. In Table 16, CLIP denotes the results reported by CLIP paper, CLIPP is our reproduced CLIP version with the original contrastive loss using exactly the same architecture on the same pre-training dataset as FILIP base- As can be seen, the FILIPbass has 6.7 points higher average accuracy than the CLIP,r rep over 12 datasets. This further verifies that the performance gain of FILIP comes from the proposed coosl-modal late interaction, rather than the data or architecture.
A.9 INFERENCE TIME OF IMAGE-TEXT RETRIEVAL
Setting. In this section, we test the inference time of both image retrieval and text retrieval on the test set of Flickr30K and MSCOCO. We compare our proposed model FILIP large against SCAN (Lee et al., 2018) and CLIP (ViT-L/14) (Radford et al., 2021). We test the inference time of CLIP and SCAN using their released code. For image retrieval, precompute the image features and report the inference time for one text query, which contains (i) the time to extract the feature of one text query, and (ii) the time of similarity calculation with all images and ranking. Similarly, for
19

---

Published as a conference paper at ICLR 2022
image
pred. (FILIP)
Patch pred. (colbert)
image
Patch pred. (FILIP)
Patch pred.
75
75
75
75
75
75
20
Bald eagle (5,6)
24
31
24
73
34
34
74
74
23
20
74
88
09
65 96
55
49
66
56
Killer whale [5,6)
28 78
64
35
76
31
65 60
68
76
3.5
29
60
75
75
61
40
75
73
40
72
19
25
48
55 25
55 55
Bee eater
(5,6)
665
15
50
75
Bullock
36
18
26
20
72
26 10
22
[5,6)
96
22
59
16 96 11
72
72 25
44
Damsalfly (5,6,7)
76
13
76
34
15
76
96
69
5:
61 29 15
51 64
Firebost (5,6)
33 64
64
33
76
13
13
13
76
71
27
60
Ski (5)
71
15
15
17
67 18 89 57
75
71
76
76
75
59
73
75
39
16
33 26
19
23
64
Necklace (5)
42
17
75
34
60
21
75
80
30
38 83
80 55
86
Mosque (5)
65
41
15
74
24
34
76
08
80
75
48
55
25
72
53
55
84
Yellow lady's
24
76
74
74
20
14
76
26
alipper (5,6,7.8)
75
65 73
75
73
Figure 4: Comparison of word-patch alignment between the proposed cross-modal late interaction and that in ColBERT (Khattab & Zaharia, 2020). "a photo of [label)." the prompt. Numbers in the parentheses after the class label indicate the location indices of the class label in the tokenized textual sequence. The correct predictions to the class labels are highlighted by opaque patches with the class label indices in red. Incorrect predictions to the padded tokens highlighted by opaque patches with the padded token indices in blue.
20

---

Published as a conference paper at ICLR 2022
Table 16: Top-1 accuracy(%)) on image classification on datasets. CLIPrep our reproduced CLIP trained with the same training data and evaluated with the same prompts as our FILIP. With the same backbone architecture, our FILIP significantly improves the zero-shot Top-1 average accuracy over 12 datasets.
CIFARIO
CIFAR100
Caltech101
StanfordCars
Flowers 102
Food101
SUN33
Aircrafts
OxfordPets
EuroSAT
ImageNet
Average
DTD
CLIP
91.3
65.1
87.9
59.4
66.7
84.4
63.2
44.5
21.2
87
49.4
63.2
65.3
CLIP
82.0
57.5
89.9
45.1
80.7
75.1
63.6
46.7
33.7
82.7
49.0
64.2
64.2
rep
FILIP
86.9
65.5
91.9
55.4
85.3
82.8
69.1
49.3
57.2
88.1
49.9
68.8
70.9
base
Table 17: Comparison on performance and inference time of image-text retrieval on Flickr30K and MSCOCO datasets.
Recall
Inference time
FOK
MSCOCO
Flickr30K
MSCOCO
R@1
image->ttxt
R655
RØ 10
R@
text->immaee R@5
RØ 10
R@1
Image->-txxt
R@5
R
R@1
text sixage
R65
R@ 10
i-to-t
L-to-i
i-to-t
I-to-i
SCAN
67.4
90.3
95.8
48.6
71.7
85.2
50.4
82.2
80.4
4.47s
7ms
21.3s
26ms
CLIP
88.0
98.7
99.4
68.7
90.6
95.2
58.4
81,5
88.1
37,8
62.4
72.2
23ms
8ms
24ms
9mns
FILIP
1000
100.0
78.9
94.4
26ms
text retrieval, we precompute the text features and report the inference time for one image query, which contains (i) the time to extract the feature of one image query, and (ii) the time of similarity calculation with all texts and ranking. The test set of Flickr30k contains 1000 images and 5000 texts, while the test set of COCO contains 5000 images and 25000 texts. The time is averaged over 1000 runs.
Results. The inference time of retrieval is shown in Table 17. Benefitting from the efficiency opti- mizations (i.e.. FP16 quantization and reduced feature dimension) in Section 3.1, the inference time of FILIP close to CLIP. In image retrieval. SCAN is slightly faster than FILIP on Flickr30K with 1000 images, because SCAN uses a lightweight GRU as the text encoder. However, SCAN is much slower than FIL.IP (i.e., about 17ms slower per query) on MSCOCO with more (i.e., 5000) images because of the slower computation involved in the two-stage stacked cross-attention when comput- ing the similarity. For text retrieval, SCAN is much slower than FILIP and its own image retrieval, mainly due to three reasons: (i) the image encoder is a Faster RCNN which is more expensive than the lightweight GRU text encoder; (ii) the text candidates are times more than the image candi- dates in image retrieval; and (ii) the similarity computation of SCAN relies the cross-attention computation, which is not straightforward to be paralleled, even in their official code; while our FILIP's similarity computation is simply matrix multiplication and is readily optimized most modern hardwares.
21


Reviews:
[{"review_id": "u7ffRJQqkGJ", "reviewer": "Reviewer_gJkL", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n(1) The proposed FILIP method achieve SoTA performance  on ITM task, which is significant for this area. \n(2) FILIP300M is a dataset of 300 million image-text paired samples collected and cleaned from the Internet, which is a huge amount of work, and will be beneficial to the whole V-L community. \n\n\nWeaknesses:\n(1) The core idea of FILIP method, namely to achieve word-patch alignment by token-wise similarity matrix through cross-modal late interaction, as far as we know, is first originated from the classical ITM method SCAN [1] (see below). So, FILIP’s cross-modal late interaction by Eqs. (3)(4)(5) is only equivalent to of SCAN’s Sum-Max (without attention) by Eqs. (9)(10), which is only a special case in SCAN’s ablation studies (see Table 3). Therefore, FILIP is lack of novelty, and we suggest the authors perform more comparative experiments about various pooling/attention/scoring methods different from SCAN [1]. \n\n(2) FILIP is not efficient in similarity matching stage, althougn it is efficient in feature extraction and embedding stage. That is because FILIP is a special case of SCAN [1], and SCAN is time consuming in cosine similarity matching, which is a commonly recognized in V-L communities. So, when the test data are very huge (for example, the real image-text search engine, usually millions of image-text pairs in total, rather than 1K/5K test data), the similarity matching time will beyond endurance. So, we suggest the authors perform some comparative experiments on the 1K/5K testing speed/time on f30k/coco with other methods like CLIP, SCAN [1] to prove their efficiency on inference phase. \n\n(3) The large-scale V-L pre-training dataset are not public/available. This might not be a very big problem since many other similar works also did this. But we cannot sure whether there are similar images/texts between the large-scale pre-training dataset (training + validation split) and the f30k/coco baseline dataset (val/test split). If the answer is true, then it is definitely unfair. \n\n(4) The SoTA performance are mainly come from 2 aspects: \n(i) The FILIP300M large-scale pre-training dataset and its cleaning/pre-processing steps. \n(ii) Grid/Patch feature (1 image, multiple features) methods like ViTs, which are the latest advances of backbone networks in CV. \nSo, it’s SoTA performance should not be largely due to the proposed cross-modal late interaction. \n\n(5) ViTs (1 image, multiple features) have the common problem of low efficiency in training/inference speed than traditional CNNs (1 image, 1 global feature). So, this shortcoming may be Inherited by FILIP. So, the training/testing speed may not exceed CLIP/ALIGN. Can the authors give more explanations? \n\n(6) The statements in this paper have a double standard on “offline/pre-computing”. \n(i) For other works: “This design complicates ... due to pre-computing and storing a large number of ROI features” (page 1). \n(ii) For their work：“FILIP successfully ... while simultaneously ... to pre-compute image and text representations offline” (page 1) \nTherefore, we don’t know weather offline/pre-computing is efficient or not? \n\n(7) There are some inappropriate expressions/mistakes in the introduction (page 1).\n(i) “One line of work” can be substitute by “Bottom-up attention [2] (see below) methods”. \n(ii) “Another line of work” can be replaced by “Cross-attention methods”. \n(iii) “self-attention (Kim et al., 2021)” is inappropriate, because ViLT is a typical cross-attention method (see Figure 3, the cross-modal Transformer Encoder), rather than (pure) self-attention. \n\nreference(s): \n[1] Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, and Xiaodong He. Stacked cross attention for image-text matching. In ECCV, 2018.\n[2] Peter Anderson, Xiaodong He, Chris Buehler, Damien Teney, Mark Johnson, Stephen Gould, and Lei Zhang. Bottom-up and top-down attention for image captioning and visual question answering. In CVPR, 2018.\n", "summary_of_the_paper": "(1) The core idea of the proposed FILIP method is to achieve word-patch alignment by token-wise similarity matrix through cross-modal late interaction by modifying only contrastive loss, which is both training and inference efficient. The authors collected FILIP300M, a large-scale cleaned image captioning dataset from the Internet for FILIP’s V-L pre-training. \n\n(2) FILIP achieves SoTA performance on ZS image classification and ITM tasks. Visualization results show its promising ability of fine-grained (visual-textual token) classification and localization. \n\n(3) To the best of our knowledge, FILIP is the first work with such high accuracy on F30k that not only does it reach double full mark of R@K (i.e. i2t[R@5 and R@10] are both 100.0), but also it achieves 580+ of rsum with only one single model (no ensemble). ", "summary_of_the_review": "The performance is very good, which mainly comes from  “big data + big model + data cleaning / augmentation + supercomputers”. The whole method is lack of novelty. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "e74sX3mA2nU", "reviewer": "Reviewer_WLou", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n1. The paper is quite well written and it is easy to follow the authors.\n2. The approach is simple yet efficient. It seems natural to extend the prior clip and align the approach with fine-grained interactions.\n3) New large-scale image-text pair dataset called FILIP300M would be valuable to the community. \n4) Experiments show strong performance on vision-language tasks.\n\nWeaknesses/Concerns:\n\n1) Model novelty is limited considering (I) similarity to  Khattab & Zaharia, and (ii) prior Vision-Language Pre-training models. \n\n2) On a similar note to the previous point, the paper states that they follow Khattab & Zaharia (2020). The changes to the proposed compared to Khattab & Zaharia seem trivial and nothing especially novel except applying to a new task. I would request the authors to discuss this in further detail to make the paper contribution clearer. The only thing the paper states 'we discard the padded tokens and use average instead summation of token-wise maximum similarities when computing the image-text alignment, which enhances the cross-modal representation learning and stabilizes training. Furthermore, we construct a large-scale pre-training dataset named FILIP300M from the Internet.'  -- This seems a minimal change. Also, where are the experimental results comparing the paper and supporting these arguments - enhanced learning and stabilized training?\n\n3) Ablation: The paper states that the use of object detector-based approaches leads to scalability issues in training vision-language models. I do not expect that to create a significant issue based on prior experience. Did other papers report also the same? Or, Did the authors' experiment and find the issue? How much impact it has?\nOn a related note, it is also important to see - How the performance changes when a faster rcnn based detector is used to understand the tradeoff?\n\n4) As the paper is interested in efficient inference, it is important that the paper include inference time for compared methods, especially in Table 2 and Table 3.\n\n5) An approach to get fine-grained interaction between visual and textual modalities avoiding the problem of pre-computing features offline), is not novel in general. As far as I am aware, such ideas are especially not uncommon in video-text retrieval. For example, prior work(Weakly Supervised Video Moment Retrieval From Text Queries, CVPR 2019) adopts a late visual-language interaction strategy with weighted pooling for video-text retrieval. However, the proposed fine-grained interaction approach seems to be new with recent transformer-based vision-language models.\n\n6) MInor: Did the authors consider - For a visual token, similarity with the full-text representation for fine-grained interaction (as a visual patch may be related to full text or multiple words, not only one word token)? Compared to For a visual token, similarity with all textual tokens (i.e., what used in the paper). Any comment on this?\n\n7) Will the dataset be publicly released?", "summary_of_the_paper": "The paper proposes an approach for large-scale vision-language representation learning that focuses on achieving fine alignment between modalities through a late interaction strategy.  Experiments on zero-shot image classification and image-text retrieval tasks show that the proposed approach is able to achieve state-of-the-art performance. A large-scale image-text dataset is also constructed from the web for pre-training.", "summary_of_the_review": "The paper is well-written with strong results. However, the proposed approach has limited novelty and some critical claims are made without strong support. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "KySx1lrRQuj", "reviewer": "Reviewer_AMdt", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* The paper is well written and the assumptions made by the authors are well motivated and described with respect to the state of the art and literature.\n* The model is quite simple and easy to implement when compared to UNITER and Unicoder-VL.\n* The results of the experiments consistently show improvement wrt other state-of-the-art methods.\n\nOn the fence (to discuss):\n1. Novelty is quite limited given that it seems to be a combination of existing models. \n   1. The use of ViT in the vision-language pre-training setup is quite interesting, thought it is not the main novelty since it has been used in other vision-language pre-training models (e.g., CLIP and ALIGN).\n   2. The most interesting contribution is to do alignment via late fusion following the method in Khattab & Zaharia (2020). The authors explain the differences wrt that paper at the end of page 4 and beginning of page 5. However, they are quite limited: using average instead of sum, excluding padded tokens, and change to contrastive loss. \n   3. Although each single piece taken independently is not very novel, I think there is some value on the combination especially in such applications.\n\n2. I understand that most of these new methods are trained on a very large scale dataset scraped from the web and it is hard to share them. At the same time, it is impressive to see that the authors have created their own cleaner version of such pre-training datasets. However, I did not see a single signal in the paper, that the authors plan to release the data or links to rebuild the data or even code. This limits the reproducibility of the experiments and also it lowers the trust on the results. I'd suggest the authors to argument on this topic and possibly commit on sharing any of the resources to being able to reproduce experiments and compare with their method.\n\n3. It is unclear why the authors run the ablation studies using YFCC100M for pre-training and not FILIP100M. It is appreciated that there is a thorough ablation study on YFCC100M. But why wasn't it carry out on FILIP100M?\n\nWeaknesses:\n\n4. Computational complexity at inference time is not clear. It seems that one can lower the computation burden of training by just using heuristics, such as keeping 25% of tokens. It makes absolutely sense to be able to do this during training, because of different epochs, randomization and large dataset. However, it is not clear what happens at test time:\n   1. Does the method extract 25% of tokens of the query and the data in the search index? \n   2. If yes to 4.1, which tokens should be chosen for the query (can be done on the fly) and which one for the search index (it should be fixed for computational reasons)?\n   3. If no to 4.2, then the claim that the method is more efficient is not valid anymore, because the comparisons to made are still quadratic as in self-attention. Anything that I am missing that makes the method more efficient for a different reason?\n   4. If we need to store the full fine-grained tokes in a search index, this means that we will have 77 tokens (as stated in the experiment section) compared to the 2 for CLIP-based methods. This means that the index is way bigger than the original one, if we were to use global features. Is this thinking correct? If yes, any idea on how to make this better?\n\nOther improvements\n* Usage of the word \"Interactive\". I am not sure why the authors used this word in the title and in other places in the manuscript. I do not see how the method requires interaction from a person or user. To me it is more about correlation or alignment between image and text.\n* Eq 3, 4, 5 are a bit confusing. They can be simplified as did in the paper of Khattab & Zaharia (2020), where you use the max(\\cdot) in eq 3 directly inside eq 4 and 5. So you do not need eq 3. \n", "summary_of_the_paper": "The paper tackles the problem of large-scale vision-language pre-training which has been quite popular recently. The main issues identified by the paper are: 1) fine-grained information might be lost due to the use of global features, and 2) alignment between text and language is typically done using cross- or self-attention, which are inefficient. The authors proposed a dual-stream method that 1) uses visual \"patch\" tokens from ViT models and 2) perform late fusion of the two modalities using a variant of the method from Khattab & Zaharia (2020). ", "summary_of_the_review": "The authors do a good job on putting together ideas from different papers and create a novel pre-training dataset which leads to new state of the art results on vision-language pre-training tasks. However, the contributions seem to be marginally significant, and I'd like to hear more from the authors about the questions I raised above. Moreover, the computational complexity claim needs to be revised and discussed more in depth, since the high price for storing the index might not compensate the small improvement in accuracy performance. If any of those two points are covered in the discussion phase, I am willing to reevaluate my score recommendation. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "1YIgSWdQ_s7", "reviewer": "Reviewer_aFb7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. Strong performance on zero-shot classification and image-text retrieval, compared with original CLIP. A fair comparison with CLIP pre-trained on the YFCC dataset.\n2. The investigation on token-level interaction to provide a more fine-grained contrastive signal in Language-Image Pre-training is interesting and meaningful for future research in this direction.\n\nWeakness:\n1. Novelty in total is limited. The proposed cross-modal late interaction is inspired by [1], but the three modifications mentioned in the paper seem to be a bit trivial. And image and text augmentation are widely used in previous works.\n2. The study on token-level interaction is not comprehensive in terms of methodology. The paper only uses max-mean late interaction without exploring different variants. For example, how will the mean-mean, max-max, or mean-max interaction work? One step further, how about we ensemble the token features inside each modality with a set of learnable weights and then compute the similarity between ensembled features?\n3. One type of experiment is missing - linear probing on the visual encoder. This is to prove the transferable representation learned by Language-Image Pre-training. \n4. The authors do not mention whether to release their collected dataset FILIP300M. If not, then it is hard to fairly compete for researchers interested in this direction. \n\n\nRef:\n[1] Omar Khattab and Matei Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In International ACMSIGIR Conference on Research and Development in Information Retrieval, pp. 39–48, 2020", "summary_of_the_paper": "The paper proposes to utilize the fine-grained alignment between visual tokens and text tokens in the contrastive loss for language-image pertaining. More specifically, the similarities between images and captions are calculated by averaging the token-wise maximum similarities. The experiments on both zero-shot image classification and image-text retrieval with different pre-trained datasets validate the effectiveness of the proposed model.    ", "summary_of_the_review": "The main drawback of this paper is that the proposed model is not that novel in terms of methodology. But this paper provides an angle to look at the token-wise interaction in contrastive Language-Image pre-training. Moreover, the performance is quite impressive and the experiments are conducted on different pre-training datasets and downstream tasks to prove the model's efficacy. Overall, I think it is marginally above the threshold. I may change my mind based on the authors' rebuttal and other reviewers' feedback.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The focus on this paper's proposed FILIP method is to perform word-patch alignment by token-wise similarity matrix through cross-modal late interaction by modifying only contrastive loss, leading to training and inference efficiency. The authors also collected FILIP300M, a large-scale cleaned image captioning dataset for FILIP’s V-L pre-training. FILIP achieves strong performance on zero-shot image classification and image-text-matching tasks, and the paper also visualizes the ability of fine-grained (visual-textual token) classification and localization. Overall most of the reviewers appreciated the idea and the generalization results, but had some concerns about not enough technical novelty over the Khattab and Zaharia Colbert paper, which this paper adopts for multimodal tasks. Some reviewers also had concerns about the dataset release but the authors promise to address this. Some reviewers were also not fully convinced about the high storage requirements and scalability for some of the retrieval tasks that the authors tested.

NOTE: The authors are also asked to describe any ethical considerations or issues that arise in their large scale dataset collection in the camera ready version of the paper, see https://arxiv.org/abs/2110.01963 for examples.
Decision: Accept (Poster)