MOFI: Learning Image Representations from Noisy Entity Annotated Images
Authors: Wentao Wu
Aleksei Timofeev
Chen Chen
Bowen Zhang
Kun Duan
Shuangning Liu
Yantao Zheng
Jonathon Shlens
Xianzhi Du
Yinfei Yang
Venue: ICLR 2024 poster (2024)
Keywords: Image representation
image embedding
image based search
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: A new vision foundation model designed to learn image representations from noisy entity annotated images, the model is strong for image retrieval and zero-shot classification tasks.

Abstract: We present MOFI, Manifold OF Images, a new vision foundation model designed to learn image representations from noisy entity annotated images. MOFI differs from previous work in two key aspects: 1. pre-training data, and 2. training recipe. Regarding data, we introduce a new approach to automatically assign entity labels to images from noisy image-text pairs. Our approach involves employing a named entity recognition model to extract entities from the alt-text, and then using a CLIP model to select the correct entities as labels of the paired image.  It's a simple, cost-effective method that can scale to handle billions of web-mined image-text pairs. Through this method, we have created Image-to-Entities (I2E), a new dataset with 1 billion images and 2 million distinct entities, covering rich visual concepts in the wild. Building upon the I2E dataset, we study different training recipes like supervised pre-training, contrastive pre-training, and multi-task learning. For constrastive pre-training, we treat entity names as free-form text, and further enrich them with entity descriptions. Experiments show that supervised pre-training with large-scale fine-grained entity labels is highly effective for image retrieval tasks, and multi-task training further improves the performance. The final MOFI model achieves 86.66\% mAP on the challenging GPR1200 dataset, surpassing the previous state-of-the-art performance of 72.19% from OpenAI's CLIP model. Further experiments on zero-shot and linear probe image classification also show that MOFI outperforms a CLIP model trained on the original image-text data, demonstrating the effectiveness of the I2E dataset in learning strong image representations. We release our code and model weights at https://github.com/apple/ml-mofi.

Published as a conference paper at ICLR 2024
MOFI: LEARNING IMAGE REPRESENTATIONS FROM NOISY ENTITY ANNOTATED IMAGES
Wentao Wu", Aleksei Timofeev', Chen Chen, Bowen Zhang, Kun Duan, Shuangning Liu, Yantao Zheng, Jonathon Shlens", Xianzhi Du, Yinfei Yang Apple AI/MI.
('wettac.wu a.t imoffev, xianzhi, www.ehhsyinfccyyeeeeee....ee wwwcccyyyyyyyyy..cco
ABSTRACT
We present MOFI, Manifold OF Images, new vision foundation model designed to learn image representations from noisy entity annotated images. MOFI differs from previous work in two key aspects: (i) pre-training data, and (fi) training recipe. Regarding data, we introduce a new approach to automatically assign entity labels to images from noisy image-text pairs. Our approach involves employing a named entity recognition model to extract entities from the alt-text, and then using CLIP model to select the correct entities as labels of the paired image. It's a simple, cost- effective method that scale handle billions of web-mined image-text pairs. Through this method, we have created Image-to-Entities (I2E), new dataset with billion images and 2 million distinct entities, covering rich visual concepts in the wild. Building upon the 12E dataset, we study different training recipes like supervised pre- pre-training. contrastive pre-training, and multi-task learning. For constrastive pre- pre-training, we treat entity names as free-form text, and further enrich them with entity descriptions. Experiments show that supervised pre-training with large-scale fine-grained entity labels is highly effective for image retrieval tasks, and multi-task training further improves the performance. The final MOFI model achieves 86.66% mAP the challenging GPRI201 dataset, surpassing the previ- ous state-of-the-an performance of 72.19% from OpenAl's CLIP model. Further experiments zero-shot and linear probe image classification also show that MOFI outperforms a CLIP model trained on the original image-text data, demonstrating the effectiveness of the 12E dataset learning strong image representations. We release our code and model weights at https://ithub..o//pppp/////
INTRODUCTION
Over the past decade, the research community has devoted significant efforts to studying the acquisi- tion of high-quality, general-purpose image representations (Donahue al. 2014 Sun al. 2017] Juan et al. 2019 Dosovitskiy et al., 2021). An effective image representation can yield impressive results on downstream tasks such as image classification and image retrieval across various domains, without requiring further customization.
Arguably, the most classical image representation learning method is based on supervised image classification (Deng et al.] 2009 [unn et al.) 2017). often using datasets like ImageNet and Ima- geNet21K (Deng et al., 2009). However, these datasets usually require expensive and difficult human labeling of precise class labels, which makes them less scalable. While some industrial labs have created large classification datasets using semi-automatic pipelines like JFT (Sun et al., 2017) or private data sources like IG hashtags (Singh et al. (2022)), how to further scale the datasets remains challenge for the research community. Another prevailing approach to learn general image represen- tations is leveraging the weakly supervised signals from text, which is easier to acquire and scale. For instance, state-of-ther -ftt--------rtt models like CLIP (Radford et al.) 2021) and ALIGN (Jia et al., (022)) learn from billions of web-mined image-text pairs using contrastive learning objective. Such pre-traine trained models can achieve strong zero-shot generalization results on various downstream tasks including image-text retrieval and image classification.
"Equal contribution. Work done while Apple.

---

Published as a conference paper at ICLR 2024
GPR1200
ImageNet-Z5, @cc@I I (%)
Dataset
Images
# Classes
Model
mAP@all (%)
ImageNet-IK
1.2M
1K
CLIP-L/140pendl
72.19
75.27
ImageNet-21K
14M
21K
MOFI-L/14
86.15
77.17
JFT-300M
300M
18K
JFT-3B
3B
30K
MOFI-H/14
86.66
78.46
IG-3.6B
3.6B
27K
(a) Comparison of MOFI and CLIP (Radford et al.
12E (Ours)
1.1B
2M
2021 on GPR1 image retrieval and ImageNet
zero-shot classification tasks.
(b) I2F and existing large image classification datasets.
Figure 1: MOFT trained on the new Peage-to-Entitier (12E) dataset, which has 66x more classes
than the previous datasets, and achieves significantly better performance on the image retrieval tasks.
Despite the great success of CLIP and ALIGN, they have not been able explore the classification objective due to the typically varying associated text for each image. However, recent studies have demonstrated that incorporating supervised data (Pham et al., 2023 Zhai et al.] 2022b) or improving data quality (Gadre et al., 2023 Cao et al. 2023) can enhance the performance of contrastive models. With these motivations mind, we (i) investigate the potential of extracting entity labels from noisy image-text pairs, and (ti) training models to learn from these extracted labels.
First, we present a simple approach to automatically label images with entities at scale. Our method leverages existing noisy image-text datasets used for CLIP training. Given image-text pair, named entity recognition model is first applied to extract entities from the text. Each extracted entity is then paired with the original image and scored by a pre-trained CLIP model, and those image-entity pairs with low CLIP scores are filtered out. The constructed dataset, termed Image-to-Entities (I2E), consists of I.IB images with 2M unique entities. To our best knowledge, I2E has the largest number of class labels documented thus far, 66 times more than JFT-3B (Zhai et al., 2022a) and IG-3.61 6B datasets (Singh et al.) (022a) (Table lb). Compared with original noisy image-text data, entity labels contain more structured knowledge, which can potentially lead to better pre-trained models.
We study different training recipes learn from the 12E dataset, including supervised pre-training, contrastive -re-training (CLIP), and multi-task learning. For the latter two, we treat entity names as free-form text and add entity descriptions of the entity to the text. The models are first evaluated on the image retrieva benchmark GPR 1200 (Schall et al. (021), and a modified image retrieval task from ImageNet. Experimental results show that the CLIP model trained on the 12E data significantly outperforms the model trained on the original image-text data. Changing the training objective to supervised classification boosts performance even more. This shows both the 12E data and the classification objective very effective for image retrieval tasks. The multi-task model reaches a new state-of-the-art of 86.15% mAP@al all on GPR 1000 beating the previous record of 72.1999 from OpenAlls CLIP model (Table Tap. We also observe a significant performance gain 2.74% on the ImageNet image retrieval task. Given its strong performance on image retrieval, we name the multi-task model MOFI, standing for Manifold OF Images.
We further evaluate the models on standard ImageNet (Deng et al. 2009) and VTAB (Zhai et al. 2020) image classification tasks MOFI trained on the 12F data performs strongly compared the CLIP model trained on the original image-text data. Specifically. for the ViT B/16 architecture (Dosovitskiy al. 202T), MOFI achieves 72.99% zero-shot and 81.32% linear probe top-1 accuracy on ImageNet, significantly outperforming CLIP by 4.27% and 1.78%, respectively (evidenced later in Table 3 and ). also achieves the best linear probe and competitive zero-shot performance on VTAB tasks, with significant improvements fine-grained recognition tasks like OxPet and OxFlowers.
Our contributions are summarized as follows.
In terms of data, we introduce Image-to-Entities (I2E), a new large-scale dataset with 1 billion images and million distinct entities, covering rich visual concepts in the wild, In terms of model training, we study various learning approaches from the constructed 12F dataset, including supervised pre-training, contrastive pre-training, and multi-task learning.
Image retrieval by default means retrieving images that are similar to a query image, which different from the image-text retrieval tasks used to evaluate CL.P. Image hased retrieval has wide industrial use cases. For VTAB evaluation, we employ the tasks from nature and specialized categories.
2

---

Published as a conference paper at ICLR 2024
Q47128 (Christmas tree)
Q871329 (Chevrolet Silverado)
Q4842083 (Bagot goat)
Q613826 (Mano del Desierto)
Q213477 (Teddy bear)
Q1152796 (Boarding pass)
Figure 2: Examples of the 12E dataset. Each caption is formatted as Entity id (Entity name)
In terms of performance, we advance the image retrieval SoTA on the GPR 120 dataset by a significant margin, and show that learning from the 12E data leads to strong image representations, with improved zero-shot performance on ImageNet and VTAB benchmarks.
NOISY MMAGE-TO-ENTITIN DATA AT WEB SCALE
In this section, we introduce our new approach construct the Image-to-Entities (I2B) dataset.
2.1 METHOD
Image-text datasets easily accessible through large-scale web crawled corpus. This has been widely used for training vision foundation models such as CLIP (Radford et al. 2021) and ALIGN (Jia al., 2021). The texts are typically gathered from webpages, e.g., image alt-text and page titles, and are often unstructured and in free-form. We build the structured 12E dataset on top of this. At a high level, the steps are: I. Construct image-text dataset from crawled web corpus; 2. Extract entities from text; 3. Entity filtering; 4. Sample the resulting dataset. Next, we describe each step in detail.
Constructing image-text dataset. Following CLIP (Radford et al. 2021) and ALIGN (Jia et al. 2021), we remove pornographic images based on a NSFW model and images whose shorter dimension less than 200 pixels from crawled web corpus. The images are further deduplicated based image hash. For each selected image, we select both alt-text and page title as text candidates. The text candidates are further filtered based on text length, full text frequency, unigram and bigram frequency. etc. Similar to Schuhmann et al. (2021), we use OpenAI CLIP-B/ model to compute image and text embeddings, and compute the cosine similarity between them and remove image-text pairs below a threshold of 0.24. From our crawled web corpus, we constructed large-scale image-text dataset which contains 8.4B image-text pairs. As one image may have multiple texts, contains 5.5B images.
Extracting entities from text. In order to extract entities from text which can be used as labels for model pre- traiiraining, we not only need to locate the named entities in the text (via named entity recognition), but also need to assign unique identifier to each entity (via named entity linking).
To do this, first find all possible entity candidate based on all n-grams and compute its probability based on its popularity from wikipedia. As entities may have ambiguity if purely based on text, e.g., Apple company vs. Apple fruit, we mitigate this problem by using other entities in the same text. Specifically, we first pre-compute an embedding for every entity as described later. For each candidate, we update its probability an iterative way until convergence.
each iteration, we first compute the embedding for the full text by combining all entity candidates' embedding based on its probability, then update the probability for each candidate based on its distance to the full text embedding. In the end, we select the best candidate with the highest probability.
Similar to Nickel Kicla (0017), , the entity embeddings are hyperbolic graph embedding trained on the Wikipedia link graph and click stream data. The loss is to minimize the distance between two
Entities be found through https:tpp... htttts:://www... wik http://///oo.oi///iiitoo..o../.

---

Published as a conference paper at ICLR 2024
0.0.0.0,,.,0....
0.1..,,,,,,00000
Shared Parameters
Inige Encoder
Encoder
Text Encoder
Image Encoder
imae Encoder
Tost Eicoderr
Image
Irage
(NI Tor, Enstt)
Image
Image
(Al Text, Entry)
Supervised
Contrastive
(CLIP)
Multi-Task (MOFI)
Figure Illustration of different approaches explored in this paper to learn image representations from 12E dataset. Supervised pre-training treats entities as labels, contrastive pre-training uses entity names and descriptions as free-form text, and multi-task pre-training combines the two.
entities occurring together on Wikipedia. For simplicity, we utilize the Wikidata knowledge base and its associated identifiers for items. This approach has reasonable precision and recall, and can be run efficiently at large scale. We are able to annotate 8.4B texts in less than 10 hours with 256 machines, each has 30GB memory.
Entity filtering. The entities extracted from text may not be always related to the corresponding image: We may not be able to disambiguate entities when there is not enough context, especially when most texts are short. Even if the entities are correctly extracted from text, it may not be always relevant the image. Take image of Honda Civic with text "Used Honda Civic for Sale in Los Angeles, CA" as an example. The text is considered as relevant based on the image-text embedding, and we can extract both are correct from the text perspective, but clearly the latter is not relevant to the image.
entities from the text, Honda Civic and
Angeles,
In order to reduce the noise in the dataset, we use CLIP model to compute the CLIP embedding for the text representation of entities, and filter out the entities which have lower cosine similarity with image embedding. To further leverage external knowledge for enhanced performance (Shen et al.) 2022), the entity names are further enriched with entity descriptions, egg., from Wikidata. Therefore, text representation of an entity formed as entity name, entity description
Sampling the resulting dataset. After filtering, the dataset contains 1.24B images with around 3. I. entities. Although it contains a large number of fine-gratd linengraieed entities, the distribution of number of images per entity is highly skewed to the popular entities (see Table [I). To ease pre-training, removed all the images associated with entities which have less than 5 images.
2.2 STATISTICS
The constructed dataset contains 1.1B images with M tities. To our knowledge, this is one the largest weakly labelled datasets available, with the largest number of entities, approximately 66 times more than JFT-3B (Zhai et al. (2022a) and IG-3.6B (Singh et al. 2022a), and around 100 time more than ImageNet-211 (Deng et al. 2009) and JFT-300M (Sun al. 2017).
Table 1: Distribution of number of images per entity, Note that entities in the [0, 5) range are removed in our final dataset.
Range
# Entities
To differentiate the constructed dataset with the original image- text dataset, we call the new dataset 12F (image-toEnntttees and the original dataset I2T (Image-to- the paper. The distribution number of images per entity is shown in Table [ It is highly skewed to popular entities. We observe that the apparel entities are the most popular ones, e,g., 0131151 (T-shirt) and 083363 (jeans) have 4.7M and 2.3M images, respectively. Examples the I2E dataset provided in Figure
955, 651
[5 10)
418, 895
in the rest of
10, 100)
1, 127, 198
100. 1000)
506, 753
1000, 10000)
117, 802
10000,
18, 768
4

---

Published as a conference paper at ICLR 2024
3 LEARNING FROM IMAGE-TO-ENTTTES DATA
In this section, we describe how the 12E dataset is used for learning general-purpose tations. We explore three types of learning approaches, as illustrated in Figure
purpose image represen-
3.1 SUPERVISED PRE-TRAINING
The image classification task is one of the simplest, but very effective methods of using labeled data for learning image representations. In our context, each entity is considered to be separate label, and the classification task is to predict which labels correspond to the given image.
There multiple choices of loss functions which can be used to promote embedding properties such as separability etc. In experiments, we use the large margin cosine loss in [Wang et al. (2118) as it was shown be simple and effective, on par with other more complicated methods (Musgrave et al., 2020). Since the 12E dataset has an immense number of entities (over 2 million), predicting all the entities in each batch is computationally costly. Similar to sampled softmax (Tensorilow Authors), a fixed number of entities is used for each batch entities of the in-batch images plus entities randomly sampled be used as additional negatives. The exact number of entities used for each batch was selected based on trade-off between performance and quality Formally,
clich Mley -mm//
Celass
Σ log KEB
(I)
eltet"
(wa)) (tet + EeEc. e wo)/tt
where eim denotes the image embedding for the k-th sample, with ck the corresponding class label. CR C²) Mcx,, C" Chatch U'random: Chutch {agle BJ. Crandom = {elc U(C') Chatch]] J. and [C[ denotes uniform distribution over classes. Size of Crandom is selected to achieve |C'| = N. m 0.15 and t are margin and temperature correspondingly. we is the embedding of class
3.2 CONTRASTIVE PRE-TRAINING
Contrastive learning of image and text correspondence is another popular way of weakly-supervised pre-training of image representations (Radford et al. [021] Jia al. 2021). Given set of image- text pairs (Ix, Tr), the goal is to learn embedding ej"
fim and ek rsl = Jint(T such that
the similarity (e Aant is larger than (ef" ,etr)) and (efm efart) for j + k. Thus, the following
cross-entropy loss is used for model training for a batch B = (Tx, Tal
k==1 Specifically,
e ,0/11//
eleh 1 ///1///
Cim contrast
Σ llog
Ctrt contrast
log
EjeBe
EyeBeloy (e y/T
KEB
KEB
Ixtxt where 7 temperature which is also learned during model training.
Lcontrast
I
ii conttast
3.3 MOFI: MULTI-TASK PRE-TRAINING
In the final setup, we combine the entity-based image classification loss with the image-text based contrastive loss to learn a universal image representation. In this setup, text embedding produced, which is compatible with the learned image representation, and can be used for zero-shot image classification, etc. Since entities are extracted directly from text, each training example already consists of triple of aligned image, text(s) and entities. Thus, it is straightforward to train the model with the above losses together. Specifically,
Leombined = XLelass + (1 Coontrrs
where is a syper-parameter to balance the two loss terms. For simplicity, we set À to 0.5 in all experiments. Note that compared with the plain alt-text used for model training, we explore the use of entity name and entity descriptions as external knowledge for better performance
Mooe details and ablation study can be found in Appendix C.
'Similar ideas have also been explored in K-Lite (Shen et al. 2022); however, the scale in K- K-Lite is much smaller (28M images compared with 1B images used in our training), while ours serves as the first evidence to show external knowledge be useful scale.
5

---

Published as a conference paper at ICLR 2024
Table 2: Results on image retrieval. mAP@all is reported on GPR1200, and kNN classification accuracy, Recall@ I, and Recall @5 are reported on ImageNet- 12T is the original noisy image-text dataset, based on which we construct our 12E dataset. Image retrieval refers image-to-image retrieval. Bold and underline are used to indicate the highest and second highest values in each bucket, respectively.
GPRI200
ImsggeNet-1K
Row
Model
All
Land
Fices
(Nat
INST
Sketch
Acqus
Acc
A
VIT-I.
Schall al 2021
IN2IK
63.2
84.9
25.3
45.0
60.4
74.8
88.5
Swin
IN2Ik+GLy2
66.2
91.7
29.5
50.3
74.85
59.2
91.1
CLIP-B/T Dopertl
Web-400M
67.33
52
32.59
51.29
ET
63.94
60.64
31,86
74.73
49.05
62.50
DINOx2-BA
B/14 (Oquab et 2013
LVM-1-2M
57.44
91.81
8.89
35.50
73.74
51.40
83:26
82.1
76.6
89.0
CLIP-B/16
76.14
84.52
41.78
90.16
83.37
95.35
97.23
53.85
90.69
86.92
79.66
71.68
86.68
8
MOFI-R/11
12B
83.33
95.45
97.24
54.45
54.16
81.02
87.67
80.27
23.34
87.12
9
CLIPAL/14
Radferd et al 2021
Web-400M
72.19
89. 88
79.96
36,85
78.52
58.67
89.34
79.77
70.14
87.32
10
DINOv2-L/14
LVM-1-2M
61.23
92.58
8.90
36.82
K2,82
59,
86.41
83.5
78.04
89.72
MOFI-1/14
85.5
12
MOFI-R/14
12R
86.67
96.80
98.62
65.52
84.0
86.82
88.05
82.73
77.60
89.34
We name the multi-task learned model MOFI, standing for Manifold OF Images, and the supervised learned model MOFIsupOnty We still use CLIP to refer to the constrastive model learned from I2E.
EXPERIMENTS
We perform comprehensive experiments to evaluate the performance of MOFI models learned from the 12F dataset. We compare with SoTA models under image retrieval (Section 4.2 and image classification tasks (both linear probe and zero-shot evaluation as detailed in Section
4.1 SETUP
We employ transformer (Vaswani al. 2017) architecture the backbone in all experiments. The default MOFI-B/16 model adopts the CLIP-B/166 (Radford et al. (202)) configuration for the image encoder, which consists of 12-layer transformer with 12 attention heads and (68-dimension hidden feature, which is projected to 512 dimension as the final image embedding. When training with contrastive objectives, we also employ the text encoder configuration from CLIP-B/16, which 12 transformer layers with 8 attention heads and 512 feature dimension. The input text is tokenized by the OPT tokenizer (Zhang et al.) 2022) with a vocabulary size of 50,265. The maximum input sequence length is set to 76.
For MOFI-L/14, the image encoder a 24-layer transformer with 16 heads and 1024-dimension hidden feature, which is projected to 512 dimension output; the text encoder is 12-layer transformer with 12 heads and 768 dimension feature. For MOFI-H/14 model, the image encoder is a 32-layer transformer with 16 heads and 1280-dimension hidden feature, which is projected to 1024 dimension as output; the text encoder is a 24-layer transformer with 16 heads and 1024 dimension feature.
All models are trained with 224x224 input image size using the AdamW optimizer (Loshchilov & Hutter 2017) with weight decay 0.1 and learning rate 0,0008, except that MOFI-L/14 uses learning rate of 0.0006. The learning rate is first warmed up to 10,000 steps, and cosine decay is applied until the last training step. Due the computation limit, global batch size 32,768, and train the other models for 1.2M steps with global batch size 16,384, so all the models have seen the same number of training examples. The number of entities N used in classification for each batch is set to 512k. All experiments are conducted using AXleam
train the CLIP models for 600k steps with
4.2 RESULTS ON IMAGE RETRIEVAL
We first evaluate the models on image retrieval tasks on GPR1200 (Schall et al. 2021 and ImageNet- IK (Russakovsky et al., 2015 GPR 200 general-purpose content-based image retrieval benchmark, which consists of subsets from six diverse domains. In total, there are 1200 categories,
httpps: t thub. com appl e e/axl earn We report additional image retrieval tasks ROxford and RParis (Radenovic al. 2018) in Appendix A.

---

Published as a conference paper at ICLR 2024
Table 3: Zero-shot classification accuracy. The top-1 accuracies (%) across ImageNet and VTAB tasks (6 tasks from natural and tasks from specialized sets) are reported.
Model
ImtgeeNet
Caltech101
CIFAR100
SVIN
DTD
OxPet
VTAB
OxFlowers
Eurosat
RESISC45
Camelyon
CLIP- CLIP-B/I6qyau B/oooa
68.38
81.59
65.66
52.09
43.24
88.03
71.72
51.50
64.68
56.89
63.94
CLIP-B/16.n-a B/16
68.72
83.91
66.88
51.69
59.20
82.31
68.30
48./1
65:27
55.23
61.59
72.94
84.38
66.89
45.98
51.59
$6.89
78.08
53,50
65.65
55.69
65.40
MGFI-8/16
90.02
78,82
52.60
CLIP-L/IAgpser
75.31
83.09
76.03
$5.68
51.75
93.02
77.80
60.92
71.27
58.17
69.81
MOFL-L/14
77.18
86.40
73.56
53.51
55.32
94,93
83.17
$1.35
64.06
50.51
68.09
MOFL-H/14
78.46
85.78
75.26
52.71
60.00
96.10
85.20
59.28
69.71
50.82
70.54
Table 4: Linear probe classification accuracy. The top-1 accuracies (%) across ImageNet and 9 VTAB tasks (6 tasks from natural and tasks from specialized sets) are reported.
Model
ImageNet
Calteah101
CIFAR100
SVHN
IXTT)
OkPer
OxHlowers
Furosat
RESISCA5
Camelyon
Avg
CLIP B/16e/rl
80.2
94.7
83,1
79.2
93.1
98.1
92.7
92.7
81.03
92.21
77.94
68.36
81.91
94.28
99.43
96.74
93.45
85.15
87.72
MOFL-B/16san
80.53
88.35
78.94
70.74
80.37
95.26
99.54
96.78
94.52
85.00
87.72
MOFI-B/16
and each category has 10 images. Follows its original paper, images are not split as query and index sets for evaluation, we retrieve the nearest neighbor for every image and use the rest as index set. We report the full mean Average Precision (mAP) @all for the entire dataset and each domain. For ImageNet, we modify its validation set to use as an image retrieval evaluation set. We randomly select one image from each category query set, and use the rest index set. For each retrieval result, we consider it as positive if has the same category as the query; otherwise, we consider it as negative. We report the top-1 and top-5 accuracy for this dataset. For kNN metric, we follow Wu et al. (2018) and train a kNN classifier on the training set ImageNet- K RRussakovsyy et al.) 2015). The best accuracy the validation over pappee-prraeterss sweep reported.
Results are summarized Table 2 We compare MOFI with existing supervised models and different versions of CLIP models, i.e., CL.P model from OpenAl trained 400M dataset (Row 3), CLIP model trained on our internal 5.5B image-text dataset (Row 4) and CLIP model trained from our 1.1B 12E dataset (Row 6). On GPR1200, MOFI outperforms existing supervised model (+17.13, Row 8 vs. Row 2) and CLIP model (+16.0, Row vs. Row 3) by a significant margin. It is interesting to see that on the Landmark domain, MOFI outperforms Swin-B (+3.75, Row vs. Row 2), which pre-trained on ImageNet216 Ik and then finetuned on clean domain specific dataset, i.e., Google Landmark V2 dataset. It is also worthwhile mention that our model performs worse (-4.5, Row 8 vs. Row 4) on the SOP domain when compared to our CLIP model. We hypothesize this is due to that using our current data mining approach, it may be hard to extract fine-grained entities from product-related text. We leave further exploration this future work. On ImageNetlk, we observe +10.84 and +4.64 improvement on Aco@I and A compared to our CLIP model (Row 8 vs. Row 4).
For the sake of interest, also compare model with DINOv2 (Oquab al. 2023), recent model trained on a curated dataset (LDV-1142) 142M) consisting of 142 million images close to domains of list of tasks. The DINOv2 model is reported with strong image retrieval performance. Table 2 shows that MOFI significantly outperforms DINOv2 on GPR1 100, but is worse on ImageNet-Ik IK. We believe this is due that DINOv2's training data is curated for target list of tasks. Apparently, ImageNet- K is included in the targeted list and other domains are not included.
4.3 RESULTS ON IMAGE CLASSIFICATION
Zero-shot evaluation. MOFI can also used for zero-shot image classification, as is trained with a text encoder using contrastive loss. Table summarizes results ImagetNet and VTAB, employing the same prompt set from CLIP (Radford et al. 2021). MOFI achieves better or similar performance compared to CLIP most datasets. For example, MOFI-B/16 improved over CLIP-B/16,wee ouu--ur- by 4.27 and 0.3 points on ImageNet and VTAB, respectively, despite the CLIP model trained on 5x larger dataset. Notably, MOFI models excel in most natural tasks within VTAB, particularly in the
7

---

Published as a conference paper at ICLR 2024
Query
CLIP
Query
CLIP
DINO
Brt Wany
IMDB
INSTRE
GLDvz
SOP
INaturalist
Oxford
IN sketch
Paris
Figure 4: Examples of top-1 retrieved images on GPR1200, ROxford and RParis evaluation
sets. Green (V) and red (X) indicate positive or negative images, respectively.
domains of OxPet and OxFlowers, where precise object recognition is crucial. Nevertheless, these models struggle with specialized tasks, possibly because the images in those tasks present greater challenges in terms of entity description. Note that zero-shot classification also requires strong text encoder. The primary goal of the MOFI model is to learn better image embeddings, and it takes significant part of the computation budget.
Linear probe. We extract image features before the linear projection to the shared embedding space following CLIP (Radford et al. (202I, and train a logistic regression classifier on top of them for each dataset. We use the Adam optimizer without weight decay. We conducted hyper-parameter sweep on training epochs (i.e., 10, 20, 40, 80, 160) and learning rates (i.e., le-1, 1e-2, le-3, le-4) to identify that MOFI performs the best, outperforming CLIP ours-121
the optimal configurations. by an average of 1.78 and 0.63 on ImageNet and VTABB respectively. When CLIP was trained using 12E data, MOFI trained with only classification objective, both models performed better than CLIP using 12T data, but worse than MOFI. This highlights the significance of a multi-task setup in achieving better performance.
4.4 ANALYSIS
In order illustrate the difference between the learned embeddings, images from different sub- categories of GPR1200 (Schall et al. 2021) well as images from ROxford and Rarii (Radenovic aL. 2018) are used retrieve the most similar images based on the embedding from different models. Results are shown in Figure 4 For each query image and each model, we show the most similar image from the corresponding index set. Images with the correct label (i.e., the same label as the query) have green frame and check-mark (V), others have red frame and x-mark(X).
Comparing the 12E and 12T datasets. We quantitatively compare the 12E and 12T datasets for CLIP and MOFI model training on image retrieval and zero-shot classification tasks. Similar to the other experiments, we put the entity name together with its original text and sample the text with equal probability when using contrastive objectives. Results are reported in Table 5 for models trained on 12T, 12F, or combined. Even for the CLIP model, switching the dataset from 12T to 12E leads to significant performance improvement classification tasks are close, with wins on ImageNet and loss on VTAB. The model trained on the combined I2E and 12T dataset is better than the model trained 12T, but worse than the model trained on 12E. These results indicate that we can also use the entity mining process as data cleaning and selection process to improve the original image-text dataset, which is aligned with the observation from Gadre et al. (202)); Cao et al. (2023).
both image retrieval tasks. The performance
the
5 RELATED WORK
Supervised pre-training extensive human-labelles datasets, such ImageNet and Ima- geNet21k (Deng et al.] (2009). has emerged as a widely adopted approach to acquire transferable visual representations. This approach has greatly expedited progress in various computer vision
8

---

Published as a conference paper at ICLR 2024
Table 5: Comparision of CLIP and MOFI models on 12T and 12E data. For image retrieval, we report mAP@all(%) on GPR1200- 200--AII, and Acc@ (%) on ImageNet-KNN classification. For zero-shot classification, we report Acc @ 1(%) on VTAB (averaged across tasks) and ImageNet. The image encoder of all models use the ViT B/16 architecture. Bold and underline are used to indicate the highest and second highest values, respectively.
Image Retrieval GPR 1200
ZS Classification VTAB
Model
Data
INAN
CLIP
12T
65.94
74.41
64.59
68.72 78.72
CLIP
12F
76.14
78.55
65.41
72.94
CLIP
I2F + 12T
73.98
77.93
64.80
73.09
MOFI
I2E
83.33
80.27
64.89
72.99
MOFI
I2E 12T
83.41
80.13
65.58
72.99
tasks like image classification (Donahue et al., 2014: Sharif Razavian et al., 2014). object detec- tion/segmentation (Girshick et 2014 Ren et al. 2015), and visual question answering (Chen et al. 2020b] [i et a 2020p. Nevertheless, the effectiveness of learned representations is fre- quently constrained by the scale and diversity of supervision within the pre-training dataset. For larger-scale pre-training, noisy labels can be also derived from noisy image-text pairs crawled from the web (Ghadiyaram et al. 2019 Mahajan et al. 2018), and certain industrial laboratories have successfully built comprehensive classification datasets by utilizing semi-automatic pipelines, such as JFT (Zhai al., 2022a), or private data sources like Instagram hashtags (Singh et al. (022a)) (Hu et al. 2023) has a similar approach as ours to utilize Wikipedia entities as a unified label space. However, they combines 14 existing dataset and primarily focus on improving generalization from known to unknown labels with respect to given text query.
Contrastive pre-training is another prominent approach for acquiring transferable image representa- tions through text supervision. In particular, models such as CLIP (Radford et al. 2021), ALIGN (Tia et al. 202T) and Florence (Yuan al. 2021) have showcased impressive zero-shot image classifi- cation and image-text retrieval capabilities by mapping images and text into a shared embedding space. In addition, CoCa (Yu al. 2022) incorporates an additional image captioning objective alongside the contrastive loss, LiT (Zhai et al. 2022c) proposes the freezing of a pre-trained image
encoder for contrastive training, and subsequent studies (Yao et al.) [021] Lee et al. [022] Li et al.)
2021 Mu et al. 2022]] Wu et al. 2021; Yang et al. 22222 Weers al.] 2023) further enhance the contrastive training objective. Furthermore, research has been conducted on various aspects including non-contrastive learning (Zhou al. 2023), the integration of external knowledge (Shen et al. 2022
masking (i et al. 2023). sparse embedding (Chen et al. 2023), and more.
Our work distinguishes itself from previous studies in two key aspects. Firstly, in terms of data, have curated a new dataset consisting of billion images and million distinct entities, making the largest dataset of its kind. Secondly, regarding model training, we propose new approach that combines supervised and contrastive pre-training, where supervised training treats entities as labels, while contrastive training utilizes entity names text and augments them with entity descriptions.
Note that the community also explore image-only self-supervised learning methods (Chen et al.
2020a He et al. 2020 Grill et al. 2020 Caron al. 2021 Chen & He] 2021 [ao et 2021 He
et al.
2022 Wei et al. 2022
We focus on learning from language supervision in this paper.
6 CONCLUSION
This paper introduces MOFI, new vision foundation model derived from billion-scale noisy entity annotated images. To train MOFI, we first construct a large-scale dataset, called Ponge-to-Entition (12E), consisting of billion images and 2 million distinct entities derived from noisy image-text pairs. Subsequently, we explore three different training approaches, and show that supervised pre-training on a large number of entities significantly enhances the performance of image retrieval tasks, and multi-task learning achieves the best performance. Additionally, demonstrate that MOFI yields robust image representations, as evidenced by enhanced zero-shot and linear probe performance on benchmarks such as ImageNet and VTAB, outperforming CLIP.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We would like to acknowledge Zhe Gan for his substantial contributions to this project, particularly for his insightful discussions regarding the experiments and thorough review of the paper. We also express our appreciation to Kushal Tayal, Madhav Sharan, and others for their foundational work in entity extraction from text. Furthermore, we acknowledge the invaluable feedback and suggestions from our teammates at Apple AIML. Special thanks go to Yang Zhao, Liangliang Cao. Xiangxin Zhu, and Vivek Rathod for their early review of the paper and for providing valuable insights and comments.
REFERENCES
Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint wxXi::216.088244 2021.
Liangliang Cao, Bowen Zhang, Chen Chen, Yinfei Yang, Xianzhi Du, Wencong Zhang, Zhiyun Lu, and Yantao Zheng. Less is more: Removing text-regions improves clip training efficiency and robustness. arXiv preprint arXiv:2305.00000 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In CVPR, pp. 9650- 9660, 2021.
Chen Chen, Bowen Zhang, Liangliang Cao, Jiguang Shen, Tom Gunter, Albin Madappally Jose, Alexander Toshev, Jonathon Shlens, Ruoming Pang, and Yinfei Yang. Stair: Learning sparse text and image representation in grounded tokens. arXiv preprint arXiv:2301. 13081, 2023.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In CVPR, 2021.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In ECCV, 2020b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jeff Donahue, Yangqing Jia, Oriol Vinyals, Judy Hoffman, Ning Zhang, Eric Tzeng, and Trevor Darrell. Decaf: A deep convolutional activation feature for generic visual recognition. In ICML, 2014.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In ICLR,R, 2021.
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, Eyal Orgad, Rahim Entezari, Giannis Daras, Sarah Pratt, Vivek Ramanujan, Yonatan Bitton, Kalyani Marathe, Stephen Mussmann, Richard Veneu, Mehdi Cherti, Ranjay Krishna, Pang Wei Koh, Olga Saukh, Alexander Ratner, Shuran Song, Hannaneh Hajishirzi, Ali Farhadi, Romain Beaumont, Sewoong Oh, Alex Dimakis, Jenia Jitsev, Yair Carmon, Vaishaal Shankar, and Ludwig Schmidt. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304.1411 2023.
Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan, Large-scale weakly- reakly-supervised pre-training for video action recognition. In CVPR, 2019.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
10

---

Published as a conference paper at ICLR 2024
Jean- Mann-Batien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a approach to self-supervised learning. NeurIPS, 2020.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. CVPR, 2022.
Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, and Ming-Wei Chang. Open-domain visual entity recognition: Towards recognizing millions of wikipedia entities. In Proceedings of the IEEE/CVF International Conference on 2023.
Computer Vision, pp.
Chao Jía, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
Da-Cheng Juan, Chun-Ta Lu, Z. Li, Futang Peng, Aleksei Timofeev, Yi-Ting Chen, Yaxi Gao, Tom Duerig, Andrew Tomkins, and Sujith Ravi. Graph-rise: Graph-regularized image semantic embedding. arXiv preprint /902. 108/4, 2019.
Janghyeon Lee, Jongsuk Kim, Hyounguk Shon, Bumsoo Kim, Seung Hwan Kim, Honglak Lee, and Junmo Kim. Uniclip: Unified framework for contrastive language-image pre-training, arXiv preprint arXiv:2209. 21412299991333, 2022.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang. Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In ECCV, 2020.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang. Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: data efficient contrastive sigguage--mage pre-training paradigm. arXiv preprint arXiv:21 10.00228,,, 2021.
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language- image pre-training via masking. In CVPR, 2023.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 100 2017.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In ECCV, 2018
Carlos Roig Mari, David Varas Gonzalez, and Elisenda Bou-Balust. Multi-scale transformer-based feature combination for image retrieval. In ICIP, 2022.
Norman Mu, Alexander Kirillov, David Wagner, and Saining Xie. Slip: Self-supervision meets language-image pre-training. ECCV, 2022.
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In ECCV, 2020.
Maximillian Nickel and Douwe Kiela. Poincaré embeddings for learning hierarchical representations. NeurtPS, 2017.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et Dinov2: Learning robust visual features without supervision. arXiv preprint 23X0v:2004.07193, 2023.
Hieu Pham, Zihang Dai. Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi- Ting Chen, Minh-Thang Thang Luong, Yonghui Wu, Mingxing Tan, and Quoc V. Le. Combined scaling for zero-shot transfer leaming. arXiv preprint arXiv:2111.1005: 2023.

---

Published as a conference paper at ICLR 2024
Filip Radenovic, Ahmet Iscen, Giorgos Tolias, Yannis Avrithis, and Ondiej Chum. Revisiting oxford and paris: Large-scale scale image retrieval benchmarking. CVPR, 2018.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS, 2015.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, al. Imagenet large scale visual recognition challenge. IJC,, 2015.
Konstantin Schall, Kai Uwe Barthel, Nico Hezel, and Klaus Jung. Gpr1200: A benchmark for general-purpose content-based image retrieval. arXiv preprint arXiv:2111.13122, 2021.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprins arXiv:2/11.02/1 14, 2021.
Ali Sharif Razavian, Hossein Azizpour, Josephine Sullivan, and Stefan Carlsson. Cnn features off dff-the-shelf: an astounding baseline for recognition. In CVPR workshops. 2014.
Sheng Shen, Chunyuan Li, Xiaowei Hu, Yujia Xie, Jianwei Yang. Pengchuan Zhang. Zhe Gan, Lijuan Wang, Lu Yuan, Ce Liu, et al. K-lite: Learning transferable visual models with external knowledge. In NeurlPS, 2022.
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens Van Der Maaten. Revisiting weakly supervised pre-training of visual perception models. In CVPR, 2022a.
Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius De Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens Van Der Maaten. Revisiting weakly supervised pre-training of visual perception models. In CVPR, 2022b.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effectiveness data in deep learning era. In ICCV, 2017.
TensorflowAAuthors Candidate sampling algorithms. URL httt:::/www.ennsoflloo. wwa.zaa/caadidate..e samps_ssppllll ing. pdf
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. NeurIPS, 2017.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Zhifeng Li, Dihong Gong, Jin Zhou, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. CVPR, 2018.
Floris Weers, Vaishaal Shankar, Angelos Katharopoulos, Yinfei Yang, and Tom Gunter. Self su- pervision does not help natural language supervision at scale. arXiv preprint axX1v:2301.07836. 2023.
Chen Wei, Haoqi Fan, Saining Xie, Chao- Yuan Wu, Alan Yuille, and Christoph Feichtenhofer. Masked feature prediction for self-supervised visual pre-training In CVPR, 2022.
Tobias Weyand, Araujo Araujo, Bingyi Cao, and Jack Sim. Google Landmarks Dataset v2 A Large-Scale Benchmark for Instance-Level Recognition and Retrieval. In Proc. CVPR, 2020.
Bichen Wu, Ruizhe Cheng, Peizhao Zhang, Peter Vajda, and Joseph Gonzalez. Data efficient language-supervised zero-shot recognition with optimal transport distillation. arXiv preprint arX1v:2112.09445, 2021.
Zhirong Wu, Yuanjun Xiong, Stella X. Yu, and Dabua Lin. Unsupervised feature learning via non-parametric instance discrimination. CVPR, 2018.
12

---

Published as a conference paper at ICLR 2024
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In CVPR, 2022.
Min Yang, Dongliang He, Miao Fan, Baorong Shi, Xuetong Xue, Fu Li, Errui Ding, and Jizhou Huang. Dolg: Single-stage image retrieval with deep orthogonal fusion of local and global features. In ICCV, 2021.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang. Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07788, 2021.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new foundation model for computer vision. arXiv preprint arXiv:2111.11432, 2021.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, Lucas Beyer, Olivier Bachem, Michael Tschannen, Marcin Michalski, Olivier Bousquet, Sylvain Gelly, and Neil Houlsby. A large-scale study of representation learning with the visual task adaptation benchmark. arXiv preprint arviv: 1010:0990.04867 2020.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR, 2022a.
Xiaohua Zhai, Xiao Wang, Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In CVPR, 2022b.
Xiaohua Zhai, Xiao Wang. Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. CVPR, 2022c.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01063 2022.
Jinghao Zhou, Li Dong, Zhe Gan, Lijuan Wang, and Furu Wei. Non-contrastive learning meets language-image pre-training. CVPR, 2023.
13

---

Published as a conference paper at ICLR 2024
Appendices
A. ADDITIONAL. IMAGE RETRIEVAL RESULTS
Table O reports results on two additional benchmarks, ROxford and RParis (Radenovic et al. which are also included in DINOv2's s target task list. On these two tasks, MOFI outperforms CLIP models by significant margin. When trained with GLDv2 (Weyand et al.) 2020) data, which is similar domain as ROxford and RPairs, MOFI (f model achieves comparable performance to DINOv2 across all metrics. It is worth to note that DINOv2 uses smaller patch size and is also distilled from a larger g/14 model. Both models are still worse than specialized models that are designed for these tasks.
2018
Table 6: Additional image retrieval results ROxford and RParis. DINOv2 is trained with more in-domain data and distilled from a larger g/14 model.
ROxford
Paris
Model
Medium
Hard
Medium
Hard
Generic models
CLIP B/16///66
36.84
12.26
67.34
44,70
CLIP-B/I 6quo
38.32
11.16
69.41
47.27
MOFI-B/16
67.33
36.93
86.08
71.75
CLIP-L/1
44.82
18.22
68.59
MOFI-//1
70.97
41.20
85.55
71.97
Generic models with in-domain data
MOFI-B/16 w/ GLDv2
69.24
39.08
88.35
76.15
MOFL-B/16 w/ GLDv2 (F)
71.05
46.24
88.39
75.88
DINOw2-B/14 (quaa al. 2025
72.90
49,50
90.30
78.50
Sperialized modils
DOLG YYnng et 2021
80.5
58.8
89.8
77.7
MSTFC (Man et M1ZZ
80.3
60.3
91.2
80.6
B. VISUALIZATION OF THE MOFI IMAGE REPRESENTATION SPACE
We show the distribution of fine-graing grained classes in GPR1200 eval set the feature space from MOFI in Figure 5 We represent each class with the average feature vector of its examples. We first reduce the feature dimension to 48, and then run t-SNE with perplexity of 20, learning rate of 50 for 300 iterations. The left figure shows that the six domains are grouped together. The imdb domain has the most concentrated distribution, as it primarily consists of face images, while the stfproduc! domain has more dispersed distribution, it encompasses wider range of diverse categories. The right figure shows the distribution of different product categories in stfproduct domain. The categories that are similar to each other located closer together in the embedding space compared to those that are not similar, e.g., coffee maker and kettle are more closely related than fan and sofa.
C. ABLATION STUDY
In this section, we perform ablation study on the number of in-batch negatives for supervised pre- training, and the number of entities needed to achieve the best performance. Limited by time and computation, we train models with 300k steps, and the learning rate cosine decay to 0 at the 300k step. At last, also compare the I2E and I2T data. We evaluate the model on two image retrieval tasks (GPR1 @GPR1200 200 and ImageNet), and two zero-shot classification tasks, including VTAB averaged across 9 tasks and ImageNet.
In-batch negatives for supervised pre-training. As described in Section 3.1 the supervised learning objective is required to predict from 2M entities, which is very computation costly and slow. To make training more efficient, a simple strategy that samples entities in each batch is used, which indicates
((I) indicates that the full image is resized to 224x224 regardless of its original aspect ratio, which is similar to DINOv2 setup. All other MOFI and CL.IP models resize the shortest edge 224 first and then perform center crop.
14

---

Published as a conference paper at ICLR 2024
giv2
Imdb
inat
sketch
Instre
stfproduct
mug
fan
bicycle
stapler
coffee maker
kettle
toaster
lamp
cabinet-
table
chair
Figure 5: I-SNE visualization of MOFI learned image representations on GPR1200 evaluation set. The left figure shows the distribution of six domains in the feature space. The right figure shows the distribution in stfproduct domain.
Image Retrieval
Zero Zero-Shot Classifiation
Image Retrieval
Zero Shot Clansification
6AP Acc
Acces
S
+ (233
IrageRie
01 212
2.
20
32th 212
213
15th
2036
13. 100L 200
a0de IN
120h 2006 JM AT 10.
(a) Ablation on in-batch entities (negatives).
(b) Ablation the number of 12E entities.
Figure 6: Ablation study on the number of in-batch negatives and the number of entities to create the I2E dataset.
N - 1 effective negatives for each loss calculation. N = 512k is used as the default setting, and Figure [6a] shows the ablation of using different number of negative entities during training. Results on image retrieval and pero-shot classification consistently demonstrate a similar pattern, indicating that the continuous addition of negative samples up to 32k leads substantial improvements in the model's performance. However, introducing additional negatives beyond 32k only brings marginal improvements, lacking significant impact.
How many entities are needed in I2E? As indicated in Table | the 12E dataset consists of 2 million entities, excluding those with fewer than 5 images. We conduct more detailed investigation by examining various quantities of entities (Figure 6b). In particular, we select the top number of entities depending on the number of associated images. We start from 20k entities which is similar to the scale of ImageNet 21 and then select top 50k, 100k, 250k, 1M, and All (2M) entities. Adding more entities consistently improves both image retrieval and zero-shot classification performances until reaching 1M entities. Adding more entities after 1M does not improve the model but also not hurt. We hypothesis the model size may not be large enough to leverage the tail entities, or the current evaluation cannot reveal the potential performance improvement brought up by adding the tail data. Thus, we keep the 2M entities in the dataset, and leave further study for future work.
15


Reviews:
[{"review_id": "Xpe0CAT5gM", "reviewer": "Reviewer_fBZj", "summary": "In this paper, the authors propose a vision-language model akin to CLIP, called MOFI. However, unlike CLIP, which primarily focuses on pure contrastive learning with image and text captions, the authors suggest supplementing this by directing the model's attention to specific entities within the captions. They achieve this by integrating an auxiliary task into the standard CLIP training objectives. To accomplish this, the authors introduce a new large-scale dataset, Image-to-Entities (I2E), containing 1 billion images and 2 million distinct entities.\n\nTo construct their dataset, the authors initially utilized a large crawled web corpus comprising 8.4 billion image-text pairs. They then applied a named entity recognition model to the text associated with the images, which could be derived from the captions or titles. However, due to the potential for multiple words to refer to the same actual entity, the authors implemented entity linking to connect entities to specific identifiers. To do so, they followed previous methodologies by learning entity embeddings based on graph embeddings from Wikipedia data. To determine the specific identifier linked with the entity, they utilized other entities within the same text to disambiguate. They combined other entity embeddings and iteratively computed the probability of the identifier based on the distance to the full text embedding. To eliminate noisy entities unrelated to the image, the authors used the image-entity distance computed by CLIP to filter out those with low similarity.\n\nThe authors explored three strategies for training models on their new dataset. The first strategy involves a straightforward classification-based objective where the aim is to classify an image as one of a fixed number of entities using a standard, fixed-size classifier. The second approach is contrastive pre-training, employing a cross-entropy loss for contrastive training, similar to CLIP. Finally, the authors proposed MOFI, which integrates both a standard contrastive learning objective and the classification-based objective by combining the two losses.\n\nThe authors conducted experimental evaluations of their approach across various tasks. Initially, they evaluated their model on the image-to-image retrieval task and exhibited significant gains compared to standard CLIP, controlling for the model backbone architecture. The authors demonstrated that, in many settings, their approach achieved state-of-the-art performance and outperformed other models overall. Next, they evaluated zero-shot classification across ImageNet and VTAB. The authors showcased that their approach achieved state-of-the-art performance on ImageNet and the VTAB Benchmark for their largest model size. Lastly, the authors evaluated the performance of linear probing with their model, illustrating improved performance over CLIP. Additionally, the authors included qualitative results showcasing examples of retrieved images from various benchmarks and a comparison of CLIP and MOFI models on image-to-text and image-to-entity tasks.", "questions": "1. What is the key benefit of MOFI's technical approach vs contrastive captioners like Coca? It seems that Coca significantly outperforms MoFi on several benchmarks significantly (e.g. Imagenet). Conceptually, the contrastive captioner seems to be more straightforward as well and doesn't require the complex entity linking that MOFI does. It seems that the entity prediction task is a subset of the captioning task that Coca addresses. Also, Coca is capturing entities AND verbs (and other words).\n\n2. Have the authors explored the statistics on how often the model predicts entities from the long tail? Is it mainly focusing on more common entities? It seems the core use of the model would be to classify an image as an entity from Wikipedia - this is an important task for knowledge graph construction and coreferencing, but it is not clear that the authors have evaluated the accuracy on that task - which seems to be the most unique part of the work.\n\n3. Will the dataset and model be released? If not, the contribution would be the technical approach, but as stated above, it is not at all clear that it is significant compared to other ways of training contrastive models (e.g. LiT) on equivalent size datasets AND it is not clear that this multi-task formulation is superior to existing methods for contrastive captioning (Coca).\n\n4. Table 2 - why evaluate on image-image retrieval when CLIP and other baselines were not trained explicitly for image retrieval. These methods have been trained for image-text retrieval, not intramodal retrieval. In contrast, MoFI's classification objective could be seen as pulling similar images together by ensuring they have a similar object distribution. It seems that if we are going to be comparing against VL models like CLIP, LiT, etc. we should focus on those types of tasks (zero shot classification, linear probing, etc.).", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "In terms of strengths, before delving into the technical details of the method, the writing of the paper is very good and appears highly polished. The figures are well-made, and overall the presentation is of very high quality.\n\nIn terms of the substance of the paper, the idea of using entities as an additional type of supervision is an interesting one. By explicitly factoring out the entities from the text and then forcing the model to perform a task on those directly, the authors essentially force the model to learn an entity-centric visual representation. In particular, rather than allowing the model to get by with some rough semantic similarity of entities which can be encoded in a feature representation for image-text matching, by explicitly forcing the model to understand millions of specific entities and to disambiguate them, this explicitly forces the model to learn representations for differentiating these very fine-grained entities. Thus, the concept of using entities as an auxiliary task is an interesting one and well motivated.\n\nThe authors create a new image-to-entities dataset, which consists of a billion images and 2 million distinct entities. As the authors point out, the dataset has the largest number of distinct class labels than any other dataset, 66 times more than the next one cited. Thus, the dataset created by the authors poses a true challenge, as classifying images into 2 million categories is a daunting task even for the most powerful models. Thus, the image-to-entity dataset could serve as a new benchmark or task for large-scale visual recognition.\n\nIn terms of experimental results, the authors demonstrate that MOFI outperforms a number of recent state-of-the-art baselines and a number of image-text foundation models, setting a new state-of-the-art for CLIP-style models on a number of different benchmarks, as they note.", "weaknesses": "Despite being an impressive and large-scale model that the authors have trained, in my view, the approach has several significant weaknesses.\n\nPrimarily, in terms of technical novelty, there seems to be minimal innovation in the proposed approach. The authors employ a standard contrastive learning approach alongside a standard classification objective, combined through a linear combination. In the context of entity extraction, the authors utilize a named entity recognizer, following an existing approach for learning entity embeddings from Wikipedia. Therefore, in terms of actual technical contribution, the paper seems rather limited. The primary contribution appears to be the extensive scale at which the model operates.\n\nRegarding entity linking, I find the proposed approach weak. It seems the authors don't conduct any form of multimodal entity linking. The process involves initial entity linking using entity embeddings on the text side, followed by a filtering step using a pre-trained CLIP model. A stronger approach for entity linking could potentially be achieved by integrating visual features into the knowledge embeddings. Since this component is central to the proposed approach, such a modification could potentially reduce the number of spurious entity linkings. I'm also curious if the authors experimented with off-the-shelf entity linkers, like BLINK.\n\nFundamentally, the authors emphasize the significance of the entity prediction task over standard contrastive learning. They argue that the experimental results demonstrate the advantage of predicting these entities. However, this claim is not entirely apparent. For instance, in table 2, the authors report better performance using their approach compared to CLIP and other baselines. Yet, this comparison might be somewhat unfair due to the substantially larger size of the author's dataset. For instance, CLIP's dataset comprises 400 million, whereas the author's dataset is 1 billion. Therefore, a comparison controlling for dataset size would be more appropriate to demonstrate the actual improvement using the author's approach rather than a mere performance increase due to a larger dataset. It would be beneficial to see a comparison with MOFI using 400 million images to understand how it compares to CLIP. Table 3 presents a more precarious situation. For zero-shot classification, we observe that the authors' multitask model doesn't show a clear advantage over CLIP on the ImageNet and VTAB benchmarks. It's not evident that the multitask formulation actually enhances performance compared to CLIP, especially considering the larger dataset size accessible to the authors. The MOFI model slightly outperforms CLIP in linear probing but only marginally.\n\nAdditionally, I'd like to ask the authors why they believe the training setting proposed in MOFI is the best approach. Since the advent of CLIP, a considerable amount of work has explored contrastive model training. For example, the LIT paper (Zhai, X., Wang, X., Mustafa, B., Steiner, A., Keysers, D., Kolesnikov, A., & Beyer, L. (2022). Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 18123-18133).) demonstrated significant performance gains by locking the image branch of the model and tuning the text model using a private dataset on which LIT was trained. There are also other large foundation contrastive models like the open-source LAION-H-14 model (https://huggingface.co/laion/CLIP-ViT-H-14-laion2B-s32B-b79K) that the authors could have compared against. These models, not using the described objectives, seem to perform similarly to the author's approach, making this comparison more equitable since the other models the authors compare to seem to have substantially less data.\n\nRegarding the approach itself, it remains unclear if the concept of contrastive learning with a classification objective on entities is necessary. Other works like Coca (Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., & Wu, Y. (2022). CoCa: Contrastive Captioners are Image-Text Foundation Models.) , demonstrate that a similar objective could be achieved by performing contrastive learning + captioning. For example, on ImageNet, the authors obtained 86.3% zero shot performance using this contrastive captioning-based approach, which significantly outperforms the authors. The contrastive captioning approach actually seems to encompass what the authors are doing in the MOFI paper, as a caption needs to be generated, not just the entity. The authors focus solely on entities, yet other crucial visual relations like events, actions, and visual properties are omitted in their approach. These could be captured under the contrastive captioning setting. Therefore, it's uncertain whether MOFI's approach is superior to training vision-language models as opposed to the contrastive captioning-based approach.\n\nIn summary, it's unclear whether the multitask learning approach presented is the optimal way to utilize this data. There's a lack of clear performance analysis against contrastive captioning-based models, making the experimental results challenging to interpret due to the differing dataset sizes.\n\nAdditionally, there's a concern regarding the technical approach itself. Filtering out entities using CLIP could potentially result in error amplification. If an entity is poorly captured by CLIP, using it to filter the dataset might exclude that entity.\n\nIt seems that the paper would have been better positioned as a very large scale visual entity linking paper, than a vision-language foundation model paper, since the key novelty the authors seem to be addressing is the ability to disambiguate these various entities. However, that itself raises a lot of questions. For example, how do we ensure that more common entities don't overwhelm the long tail of entities? This does not seem to be explicitly handled by the authors' approach and currently authors do not perform an evaluation sufficient to consider the paper for the visual entity linking task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0kAHud5HYp", "reviewer": "Reviewer_FYZT", "summary": "This manuscript presents two main contributions. Firstly, it constructs a large-scale I2E dataset in which the correspondences between images and entities are collected. The dataset is derived from a large-scale web corpus, and the authors enhance its data quality by cleaning and filtering. Secondly, the manuscript explores various training recipes and ultimately adopts a multi-task learning strategy that combines supervised learning and contrastive learning. The multi-task model is trained on the constructed I2E dataset to perform image retrieval and classification tasks. Through experiments, the manuscript demonstrates that both the dataset and the training method contribute to achieving better results across multiple downstream tasks.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1.\tThe manuscript constructs a large-scale and high-quality I2E dataset, which can be utilized for model training across various downstream tasks such as image classification and image retrieval. Furthermore, through a comparative analysis between the original CLIP, CLIP trained on I2T, and CLIP trained on I2E, the manuscript demonstrates that the constructed I2E dataset improves model performance.\n2.\tThe experiment is comprehensive, conducting thorough comparisons to assess the impact of different datasets on model performance, thereby validating the high quality of the dataset. Additionally, through comparisons between different models, the manuscript verifies the effectiveness of the proposed training approach.\n3.\tThe paper is well-organized and clearly written.", "weaknesses": "1.\tThe number of training data can be shown in Table 2 for more clear comparison.\n2.\tIncomplete experimental comparison in Tables 3 and 4. Why is there only CLIP for comparison? Other methods for solving zero-shot/linear probe classification should be discussed and compared.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZD3EFSmRV7", "reviewer": "Reviewer_4Nss", "summary": "This article primarily presents a billion-scale Image-to-Entities dataset, which includes 1 billion images and 2 million distinct entities. Based on this dataset, the authors have attempted a series of model training algorithms, including supervised learning, contrastive learning, and a combination of both in the form of multi-task learning. The constructed models have achieved state-of-the-art performance on the GPR1200 and VTAB datasets.", "questions": "I am more concerned about how MOFI performs on existing hot tasks, such as whether it is more advantageous to combine with LLM than CLIP, or whether MOFI can be used for image generation tasks.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ A large-scale image-to-entity dataset is provided, which has the potential to be reused by other pre-training models for vision-language related tasks.\n+ Based on this data, a pre-training model has been trained, which can serve as a general-purpose tool for tasks such as image classification and image retrieval.", "weaknesses": "+ There are a few unfair comparisons between MOFI and CLIP: (1) CLIP can handle text at the sentence level, serving as the base model for MLLM image inputs. It can be used in tasks like stable diffusion in text-guided image generation. (2) MOFI only utilizes the proposed datasets, which definitely excels at mapping images to entities. The selected dataset for the experiment also leans towards examining the correspondence between images and entities.\n\n+ The training of the MOFI model is limited to image-level correspondence to entities. However, in reality, a single image may contain multiple entities, necessitating the inclusion of region-level correspondence in the model's training. Unfortunately, MOFI does not take this aspect into consideration.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3p0iVARIxZ", "reviewer": "Reviewer_NPAD", "summary": "This paper proposes a new vision foundation model, uses three different training strategies, and verifies the performance on the image retrieval task on the constructed dataset. Experimental results on the constructed large-scale data set verify the effectiveness of the proposed new model and training strategy.", "questions": "I am wondering why using entity filtering images for multi-task pre-training helps the retrieval problem. This paper could benefit from illustrating more rationale behind using Image-to-Entities dataset.\n\nWhat are the major difference between multi-task pre-training methods employed in this paper and the contrastive learning?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "It looks novel and promising to leverage  Image-to-Entities data to improve image retrieval performance using multi-task pre-training.\n\nAblation studies are pretty solid and comprehensive to reveal characteristics of the proposed method.\n\nThe analysis of different learning strategies is interesting. The paper provides a nice insight into exploiting image representation learning strategies for a particular task.", "weaknesses": "In the experiment section, my major concern is that it lacks comparisons with SOTA methods. Ideally, it is encouraged to have comparisons with SOTA methods from both traditional methods and methods based on foundation models. So it will show improvement and advances of the proposed method in this area.\n\nA lack of analyses on efficiencies. In particular, the performance gains from the method should be evaluated along with its time and memory complexity. Does the complex the model, the higher the performance?\n\nThe authors observe a similar performance gain on the ImageNet image retrieval task. Why does model performance improve roughly the same on both ImageNet and Image-to-Entities ? The author may want to clarify rationale behind their observations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)