Classification Done Right for Vision-Language Pre-Training
Authors: Zilong Huang
Qinghao Ye
Bingyi Kang
Jiashi Feng
Haoqi Fan
Venue: NeurIPS 2024 poster (2024)
Keywords: Text Supervision
Visual Models
Pretraining
classification
Primary area: other

Abstract: We introduce SuperClass, a super simple classification method for vision-language pre-training on image-text data. Unlike its contrastive counterpart CLIP who contrast with a text encoder, SuperClass directly utilizes tokenized raw text as supervised classification labels, without the need for additional text filtering or selection. Due to the absence of the text encoding as contrastive target, SuperClass does not require a text encoder and does not need to maintain a large batch size as CLIP does. SuperClass demonstrated superior performance on various downstream tasks, including classic computer vision benchmarks and vision language downstream tasks. We further explored the scaling behavior of SuperClass on model size, training length, or data size, and reported encouraging results and comparisons to CLIP. https://github.com/x-cls/superclass

Classification Done Right for Vision-Language Pre-Training
Zilong Huang
Qinghao Ye
Bingyi Kang
Jiashi Feng
Haoqi Fan
ByteDance Research
Code & Models: x-cls/sueesclass
Abstract
We introduce SuperClass, super simple classification method for vision-language pre-training on image-text data. Unlike its contrastive counterpart CLIP [57] who contrast with text encoder, SuperClass directly utilizes tokenized text supervised classification labels, without the need for additional text filtering or selection. Due the absence of the text encoding as contrastive target, SuperClass does not require a text encoder and does not need to maintain a large batch size various
CLIP [57] does. SuperClass demonstrated superior performance downstream tasks, including classic computer vision benchmarks and vision lan- guage downstream tasks. We further explored the scaling behavior of SuperClass on model size, training length, or data size, and reported encouraging results and comparisons CLIP
Introduction
Pretraining methodologies [35, 57, 51, 60] that directly harness web-scale image-text dataset have transformed the field of computer vision in recent years. Among them, contrastive language image pretraining (CIJP) [57] has gained escalading popularity and become predominant due to the follow- ing reasons. First, it serves the industry-standard pre-trained model that facilitates zero-shot visual recognition [50, 52] and finetuning on downstream tasks [19, 17]. Second, proper scaling behav- iors [12] are observed such that CLIP can consistently benefit from larger models and bigger data to some extent. Third, it offers strong cross-modal abilities as it is inherently designed to understand and connect information across text and images. Therefore, CLIP-style models are the default choices for most modern Visual Language Models [47, 2, lJ, which connect a vision backbone with a deep language model [69, 13].
Despite its success, CLIP necessitates very large batch sizes for ttyicaly-y over 64,000-to achieve optimal performance, along with substantial computational resources for text encoding. This high computational demand limits accessibility for researchers with limited resources and engineering expertise. In our work, we aim address the heavy computational burden by replacing contrastive methodology with simpler classification approach, eliminates the need for large contrastive batch sizes, and text encoders.
In this work, we revisit the classification method for pretraining on large-scale text-image pairs. Some previous works [54, 31, 39, 27, 51] attempt to tackle this by employing bag-of-worrd classification in a weak supervised learning manner. However, most of these studies have been conducted on small scale, and there is no evidence demonstrating their scalability in terms of data and model size. In contrast, our method demonstrates the performance of SuperClass on a scale comparable to CLIP [57], achieving favorable model performance with 13 billion seen samples unique text-image pairs. Some other concurrent efforts [31] have also attempted to replace contrastive learning with classification. However, they rely heavily on preprocessing the text modality, using
I billion
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

InfoNCE
text tokens
image
text
image
image
text tokens
image
takens
tokens
CLIP
SuperClass
Figure 1: (left) CLIP uses separate Transformer encoders extract vector representations from image-text pairs. The text encoder operates on a subword-level tokenizer. (right) The proposed bag of subwords classification only uses the single Transformer encoder.
bag-of-words and other hand-crafted rules to convert text into semi-labels. Some common practices include filtering, word segmentation, lemmatization, and the removal of numbers and stopwords to create a unique vocabulary of clean words. We found the preprocessing often eliminates long-tailed words stopwords that contain valuable information for representation learning (see Sec. 4.4). In contrast, SuperClass simply utilizes raw word tokens as supervision signals without requiring any hand-crafted preprocessing: filtering removal of stopwords. Hence SuperClass preserves all information from the original text descriptions as supervision signal.
We proposed Super-simple- Chappl--Cassiiicttion approach (SuperClass) that simply trains classify raw text tokens and scales as good as CLIP. As shown in Figure I, similar to CLIP, SuperClass directly operate on text tokens with any manual text filtering. Our comprehensive empirical study shows that even without the need for a text encoder, classification methods can achieve performance comparable to the contrastive approach in terms of both model capabilities and data scalability. We demonstrate that SuperClass is a competitive alternative to its contrastive counterpart on both image classification and vision & language tasks. Pretrained the same Datacomp-I IB [21] datasets with an equal number of seen samples, SuperClass dominantly outperforms its contrastive counterparts across various of vision only and vision & language scenarios. We further explore the scaling behavior of SuperClass concerning model size and number of seen samples. Experiments suggest that classification based methods can exhibit competitive or even superior scaling behavior compared to their contrastive counterparts. We hope our method, experiments and analysis can encourage future potentials of classification-based methods the foundational vision-language pretraining methods.
2 Related Work
With the growing availability of large-scale, web-sourced image-text datasets [57, 65, 6, 68, 21, 63, 62], new methods have emerged leverage this data as supervision for training deep representa- tions. These approaches typically involve one of three strategies: using text as classification labels, implementing image-text contrastive learning, or treating text as autoregressive targets.
Text as classification labels. The exploration of image-text data for model training has deep roots, with early work like Mage-to-Woor[544 over two decades ago aiming to enhance content-based image retrieval. This study pioneered efforts to train models to predict nouns and adjectives in text documents linked images. Building on these early ideas, subsequent research has sought to improve data efficiency[6s 43], model effectiveness [31, 27], and vocabulary expansion [27, 78, 51, 39]. With the recent develop of network architecture, Tag2Text [27] and RAM [78] have employed Vision Transformers (ViT) [18] as vision backbones, extracting nouns from the CC12M dataset [6] and, through a combination of rules and manual selecting, arrived at 6,449 words to use as classification
categories. Similarly, CatLIP [51] has filtered out "gold labels" from the CC3M [65] and Datacomp-
1B [21] datasets based on certain rules, and then trained visual models using even larger-scale image-text pair datasets.

---

Unlike previous classification methods that rely on complex rules or manual filtering to curate "gold labels" for classification vocabularies, our approach eliminates the need for such filtering. Instead, we directly leverage text tokens as classification categories, preserving valuable textual information that might otherwise be discarded.
Image-text contrastive learning. Large-scale contrastive vision-language pretraining gained trac- tion with the introduction of CLIP [57] and ALIGN [30]. Since then, numerous approaches have focused on enhancing CLIP's performance [76, 45, 44, 11,7, 7, 74]. For instance, SigLIP [76] reduces the computational load of CLIP's softmax-based contrastive loss by employing a sigmoid loss for local pairwise similarity calculations, LiT [77] adopts pretrained vision and language backbones with contrastive training, while other methods [45, 44, 19] aim to enhance training efficiency in image-text pretraining. Intern VL [11] further innovates by integrating a large language model as the text encoder within CLIP.
In our approach, we challenge the necessity of an additional backhone to encode text information for contrastive learning. Instead, directly use text token input the supervisory signal, eliminating the need for text encoding and avoiding the computational overhead large contrastive operations. This streamlined setup achieves performance comparable to dual-backbone methods.
Text as autoregressive targets. Various recent studies [15, 61, 38, 32, 41, 26] have delved into employing image captioning for model pretraining. SimVLM [71] has innovated this field by pioneering the pretraining of a multimodal encoder-decoder that fuses vision and language at an early stage, leveraging hybrid architecture for applications such visual question answering (VQA). CapPa [70] demonstrates that a simple encoder-decoder setup can efficiently pretrain vision encoders solely through captioning. Furthermore, recently studies [75, 42, 37] combine contrastive learning with captioning objectives, occasionally incorporating an additional text encoder.
In this work, we revisit the classification approach using large-scale visual-language datasets. Unlike the image captioning methods mentioned carlier, our classification method integrates the text captioning decoder within the vision encoder, allowing single vision encoder to connect both modalities. Experiments demonstrate that SuperClass achieves competitive, and often superior, performance across various downstream tasks.
3 A simple classification approach to pretrain vision encoders
In this section, present proposed approach, SuperClass, which employs classification- based pretraining method using text supervision. We begin by outlining the general framework of SuperClass. Next, we explain how text is converted into category labels without the need to select "gold labels", allowing all text to supervise the training of the image encoder. Finally, we illustrate our choice of loss design among various classification losses. Additionally, recognizing the differing significance and discriminative power of each word, we incorporated inverse document frequency as class weights in the loss design.
Overview. We aim to establish a pretraining method based on image classification that matches CLIP in simplicity, scalability, and efficiency. To achieve this, we follow the standard protocol by utilizing Vision Transformer (ViT) backbones as vision encoders, followed by a global average pooling layer and a linear layer as the classification head to output the logit vector x. The supervision targets are derived from the text associated with the image, and the classification loss computed using the text-derived classification labels and the predicted logits.
Texts as Labels. We directly use tokenized text K-hot labels, where K is the number of tokens
in the given sentences. More specifically, for a given image-text dataset D = {(I, T:) i [1, N]} with N pairs of images I and text captions T. we differ from previous classification-Dasee based methods by directly using an existing subword-level tokenizer, such as the one used in CLIP or BERT, with a vocabulary size V. This tokenizer inputs the text and obtain the set C of corresponding subword IDs, which serves as the classification labels. The label in the set C satisfies {c [1, VJ). The classification labels C will be converted into K-hot vector y E TR. where y. = 1 when e in the set C, otherwise ye 0.
3

---

Compared to previous methods, our approach does not require any preprocessing manual threshold setting, making it straightforward. At the same time, also avoids the out-of-vocabulayy issue that might be encountered by previous approaches.
significant body research has focused on multi-label classification loss. However, it is important to emphasize that our primary goal is pretrain vision encoders rather than prioritize multi-label classification accuracy. multi-label scenario, a Softmax loss applied in SuperClass by representing labels in probabilistic manner, where De normalized weighted label.
Classification Loss.
V
lee = Σ De log
E ed
We evaluated several multi-label classification losses, including Softmax loss, BCE loss, soft margin loss, ASL [59], and two-way loss [33]. Surprisingly, the simple Softmax loss yielded the best pretraining results. This may be due to the fact that existing multi-label classification losses assume that labels are precise and exhaustive, aiming to optimize the margin between positive and negative classes. However, the inherent noise in image-text data and the limitations of text in fully capturing an image's content mean that not all objects an image are always referenced in the accompanying text.
Inverse Document Frequency. Within the subword vocabulary, not all categories contribute semantically equally, different words carry varying amounts of information. Additionally, the subword dictionary contains many words unrelated to visual content that frequently appear in sentences, which do not provide effective supervisory information. Therefore, words with higher information content should be given greater weight during training. To achieve this, we employ Inverse Document Frequency (IDF) as a measure of information significance. The fewer the number of samples containing a specific word, the stronger its ability to differentiate between samples. We use the IDF statistic of each category (subword) the weight for the corresponding classification label, assigning different weights uc to the classification labels c.
WeYe
D
Dc
We log
(2)
Eer We
1 df(c)
where [D] denotes the total number of image-text pairs, df((c) is the document frequency (df) of subword in other words, it's the count of texts containing subword For greater ease of use, have implemented an online IDF statistic that is computed during the training process, eliminating the need for pre-training offline statistics. This approach enhances the frenfdiinnsssess and portability of
our method.
4 Experiment
4.1 Experiment setup
We use standard subset of the datacomp dataset [21] for pre-training, which contains about 1.3B image-text pairs. A batch size of 16k and 90k are adopted for our classification models and CLIP models. In the ablation section, all experiments are conducted with batch size of 16k. To make fair comparsion with the CLIP, we use 90k batch size, adopt the Adam W with a cosine schedule, and set the same learning rate and decay as CLIP.
4.2 Evaluation protocols
In order to better highlight the effectiveness of pretraining method, we concentrate on the properties of the frozen representations.
Linear probing We evaluate the classification accuracy when using the full ImageNet-1k 1k [14] training set to learn a dense projection layer and frozen the parameters of backbone. We follow the linear probing training recipe from MAE [24].

---

Table I: Comparison of the Linear probing top-1 accuracy ImageNet- 1K dataset.
ViT-Base
ViT-Large
Method
PreTingning data
#Seen Samples
Top-1 1 (%)
#Seen Samples
Top-1 (%)
contrastive clustering based
MoCov3 [10]
INIK
400M
76.7
400M
77.6
DINO [5]
INIK
512M
78.2
iBOT [80]
IN22K
400M
79.5
256M
81.0
DINOv2 [55]
LVD-142M
2B
84.5
reconstruction based
BETT [3]
D250M+IN22K
IB
56.7
IB
73.5
SimMIM [73]
INIK
1B
56.7
CAE [8]
D250M
70.4
78.1
MAE [24]
INIK
2B
68.0
2B
75.8
vision-languin language pretraining based
Openai CLIP [57]
WTT-400M
13B
78.5
13B
82.7
Cappa [70]
WehL--IB
9B
83.0
OpenCLIP [29]
Datacomp-1D
13B
83.9
SuperClass
Datacomp- IB
78.7
1B
82.6
SuperClass
Datacomp- IB
13B
80.2
13B
85.0
Table 2: Performance of frozen visual represen- tations on different classification datasets. 10- shot linear evaluation accuracy on the pre-logit representation. *results from the paper.
Table 3:
Zero-shot Top-1 acc. and CIDEr
are tested on ImageNet- k dataset and COCO captions, respectively. The zero-shot accuracy SuperClass is obtained after lock-image tun- ing [77].
Method
ImageNet
Pets
Cars
Case
Backbone
O-shot
CIDEr
MAE
44.0+a.1
57.7+00
32,5=0.1
DINOv2
77.0=a.
94.2=a.1
76.8202
Openai CLIP
VIT-L/14
75.3
113.5
CapPa*
70.610.2
92.620.5
92.2
OpenCLIP
VIT-L/14
79,2
OpenCLIP
75.610.1
92.2.106
92.7.103
CLIPrrimmll
ViT-L/16
79.0
112.6
SuperClass
77.2=0.
94.6=001
92.6-10.1
SuperClass VIT-L/16
79.7
113.0
10-shot classification Following the setting of Cappa [70], we perform 10-shot classification on ImageNet-Ile 1k [14], Pets [56] and Cars [34]. For each dataset and model, run 3 times, and report the mean results and variance.
Locked-image Tuning Locked-image Tuning (LiT) [77] employs contrastive training align locked image and unlocked text models. Generally, LiT is an efficient way to equip any pretrained vision backbone with zero-shot classification and retrieval capabilities. We follow the setup from LiT[77] and assess the zero-shot classification accuracy on ImageNet- 1k [14].
Collaborating with language models Motivated by recent works [47, 11, 1, 67, 2, 71] combining pretrained vision backbones [57, 19, 76] and language models [69, 13], we investigate the amenability of the learned representations to interface with text decoder. Here, we evaluate two ways to collaborate with large language models. following ClipCap [53]. we frozen both pretrained image encoder and pretrained 12-layer GPT2 decoder [58], only train an adapter to connect the image encoder and language model to perform image captioning COCO captions [9]. 2) following LLaVA [47] setup, we train and finetune a projection layer and a pretrained large language models, Vicuna-7B [13] to solve downstream tasks, including VQAv2(val) [22], GQA [28], VizWiz(val) [23],
T-VQA(val) [66], SQA(img) [79], MMBench(en) [48], MME [20]. POPE [46], MMMU [25] and SEEDBench [40].
4.3 Main results
Comparison with different types of pretraining methods In Table 1, we compare the models trained by SuperClass with the different types pretraining methods, including contrastive or clustering based methods [10, 5, 80, 55], reconstruction based [3, 73, 8, 24], and vision-languaee
5

---

pretraining based methods [57, 70, 21J. In general, the proposed method achieves best performance among these pretraining methods.
Compared to the current SOTA self-supervised model DINOv2 [55], our method achieves a 0.5% higher accuracy in IN-1K linear probing (85.0 vs 84.5) without a bunch of bells and whistles. Although SuperClass has seen more samples, it operates as a simpler classification framework and does not employ MultiCrop, Masked Image Modeling, or Contrastive learning, as DINOv2 does. Although comparing a self-supervised learning method to a (weakly) supervised learning approach system comparison, we still observe that a simple classification pretraining approach demonstrates superior performance across many classy benchmarks that shown in Table 2.
Compared to the contrastive counterparts CLIP [57], our method achieves higher linear probing top-1 accuracy on ImageNet- IK dataset with ViT-Base (80.2 vs 78.5) and ViT-Large (85.0 vs 82.7) as backbone. For fair comparison, a ViT-Large model with a batch size 90k based on Datacomp- IB dataset. Our method consistently outperforms OpenCLIP by large margin (85.0 83.9). In Table 2, our method surpasses OpenCLIP on IN-IK and Pets by clear margins with improvements of 1.6 and 2.2 points, while being comparable with OpenCLIP on Cars (92.6 v.s 92.7).
further make comparison with OpenCLIP [29] which trains
Further comparison with CLIP In Table 3, we compare the models trained by SuperClass with the currently widely used CLIP models. including zero-shot classification, and COCO captioning. To verify the effectiveness of the pretraining method, we adapt the standard ViT [18] structure as the visual backbone and added classification head on top of it. We use the open-source Datacomp- IB [21] dataset and encounter 13B samples in the training process.
For better comparison with the CLIP models, select three types: OpenAl's CLIP ViT-L/14 trained on the internal WiT-400M data, and Laion's 's CLIP ViT-L/14 trained on the open-source Datacom-IB IB dataset. The checkpoints of these two models have been open-sourced. The checkpint of Laion's s openCI.I is downloaded from Hugginface Hub For a fair comparison, we trained the ViT-1/16 model with batch size 90k based the our codebase and Datacomp- IB dataset.
With Lock image Tuning [77], the trained classification model also gains the ability of zero-shot classification. Our method achieves 79.7% Top-1 zero-shot accuracy on ImageNet-Ik Ik datatset which is much better than OpenAl CLIP ViT-L/14 and OpenCLIP ViT-L/14, Although maybe they are not directly comparable, this do reflect that the vision model trained by the proposed SuperClass is with strong visual perception capabilities.
Combining the frozen vision encoder with a frozen pretrained 1 12-layer GPT-2 decoder [58] via trained adapter, the models are trained on COCO captions [9] and CIDEr socres are reported. We observe that the CIDEr socres of our method are slightly below OpenAl's CLIP, which may be due to the use of different datasets; OpenAl's CL.IP utilizes an internal dataset, WiT-400M. However, our approach outperforms our implemented CLIP model with the same settings.
Overall, the models trained by proposed SuperClass demonstrated marginally improved accuracy in both classification capabilities and the vision & language task when compared to the contrastive pretrained CLIP models.
Large multi-modal models Many large multi-modal models integrate pre-trained vision backbones with large language models. We explore how amenable the learned representations are interfacing with a text decoder. Following the LL setup [47], we combine frozen CLIP models and SuperClass models with the pretrained Vicuna-7B [13] and perform downstream tasks. In Figure 2, we show some results of vision&language downstream tasks. The results demonstrate that SuperClass models could achieve better performance than CLIP models on the majority of datasets. It is worth mentioning that, in comparison to CLIP models, SuperClass models exhibit significantly better performance on VQAv2 [22], T-VQA [66], and MMBench [48], which pertain to OCR and fine-grained recognition tasks, respectively. In addition, the overall accuracy measurement on VizWiz [23] are not stable due to significant portion of questions being labeled as unanswerable. To ensure the completeness of our findings, we still present the results on this dataset.
Due to space limitations, detailed numerical results are provided in the appendix. Additionally, you can find more experimental results in the appendix.
LLaio's 's CLIP https://tuggingffeccoooooooooooo ortpapaeeeee......ooteeeeeetooon XL-ssla-ooooop---------

---

ImageNeet O-shot
ImageNet Linear Probing
VQAu2
T-VQA
70
65
75
68-
25
60
70-
66
20
55
65-
64
15
50
B
60
B
5
6 FLOPs
5
FLOPs
FLOPs
FLOPs
ImageNeet O-shot
ImageNet Linea Probing
VOAv2
T-VOA
75
02-
72
35
70
70
30
78
68
25
60
74-
66
20
55
64
15
128M
512M
1,288
128M
512M
1.288
128M
513M
1.288
120M
512M
1.288
Training Samples
Training Samples
Training Samples
Training Samples
SuperCless
Figure 2: Zero-shot classification accuracy and linear probing accuracy on ImageNet- Ik dataset (left two columns); Performance of VQAv2 and T-VQA with LLaVA training recipe (right two columns). Top row: We compare the performance of vision backbones-ViT--y via classification and contrastive methods with the same batch size of 16k and 512 million samples, focusing on their size and computational cost. SuperClass demonstrates better scaling on zero-shot classification and VQAv2, T-VQA tasks. Bottom row: Comparing SuperClass and CLIP, performance increases with more training examples, mirroring the effects of model scaling. All methods are trained the same batch size of 16k and ViT-L/16 as backbone.
B/16, and 1/16 6-preerriieed
Model scaling results In the top row of Figure 2, we showcase the performance across classification and vision & language tasks for varying model scales. For a fair comparison, both CLIP and SuperClass models undergo training with identical settings, which include batch size of 16k and 512 million seen samples. As shown in Figure 2, with the model scaling up, we observe a corresponding enhancement in performance, whether it is for classification tasks the downstream tasks associated with LLaVA. Generally speaking, with the same model size, models pre-trained using SuperClass exhibit superior precision compared to those trained with CLIP. SuperClass demonstrates better scaling on zero-shot classification and VQAv2, T-VQA tasks.
Data Scaling results In the bottom row of Figure we showcase the performance across classi- fication and vision & language tasks for varying seen samples. For fair comparison, both CLIP and SuperClass models undergo training with identical settings, which include batch size of 16k and V as backbone. Figure 2 illustrates that as the number of seen samples grows, there is noticeable improvement in performance for both classification and downstream tasks linked to LLaVA. Typically, models pre-trained with SuperClass outperform those trained with CLIP in terms of accuracy when given the same amount of seen samples. SuperClass exhibits the same or slightly better scaling behavior compared to CLIP on downstream tasks. In addition, SuperClass does not require a text encoder, it offers better efficiency in training compared to CLIP.
4.4 Ablations
To verify the rationality of the SuperClass, conduct extensive ablation experiments that pretrain on datacomp-1B [21] and evaluate on several downstream tasks with different settings for SuperClass.
Word-level tokenizer vs. Subword-level tokenizer Table 4 presents the results of word-level tokenizer and subword-level tokenizer on serval daownstream tasks. We use the tokeinzer in openai CLIP as subword-leve tokenizer. We compare it with the word-level tokenizer used in CatLIP [51], which carefully selected approximately 40,000 "gold labels" from the datacomp-15 IB dataset. Aside from the tokenizer being different, all models are trained under the same settings.
For VIT-S/16, word-level tokenizer achieves better classification accuracy than subword-level tok- enizer. A possible reason is that when the model capacity is limited, the filtered clean supervisory
7

---

Table 4: Word tokenizer vs. Subword tokenizer. The performance of classification and LLaVA downstream tasks with different tokenizers. SuperClass use a subword-level tokenizer to map text into category labels. All models trained in the same settings with batch size 16k and 512M seen samples.
Classification
Vision & Language Downstream Tasks
MMBBnch
VQAv2
VizWiz
T-VQA
MMMU
GQA
SQA
MME
POPE
SEED
Tokenizer
VIT
LP
ZS
Word
S/16
68.4
53.2
65.28
55.38
43.12
15.84
65.83
49.14
1228
80.32
36,6
50.54
Subword
S/16
67.9
52.8
65.54
55.95
46.23
16.46
65.64
48.53
1306
81.02
33.2
50.43
Word
B/16
76.1
61.4
67.72
57.12
46.20
20.98
65.79
54.63
1296
81.88
34,4
53.38
Subword
B/16
76.0
61.7
68.34
57.43
41.79
24.41
65.54
54.03
1324
82.28
36.9
52.88
Word
I/16
80.3
68.2
69.47
57.36
51.94
23.87
65.00
56.27
1335
83.28
35.4
53.64
Subword
L/16
80.5
69.0
70.40
58.16
51.48
29.83
67.72
59.45
1373
84.04
36.3
53.74
Table 5: The performance of classification and LL downstream tasks with different subword-level tokenizers. All models are trained in the same settings with a batch size 16k, 512M seen samples and ViT-L/16 as Backbone.
Classification
Vision & Language Downstream Tasks
MMBench
MMMU
VQAv2
VizWiz
T-VQA
MME
POPE
SEED
GQA
SQA
Tokenizer
Vocab
LP
ZS
OpenaiCLIP 49.152
69.0
70.40
58.16
51.48
29.83
67.72
59.45
1373
84.04
36.3
53.74
WordPiece 32,000
80.5
68.5
69.95
57.76
49.07
29.33
65.99
56.18
1375
83.37
35.1
54.05
SentencePicce:32.000
80.2
67.8
69.52
57.95
49.20
28.56
65.05
57.47
1301
82.16
34.8
53.52
information may be more conducive model convergence. However, with the increasing size of the model, subword-level tokenizer gradually outperforms the word-level tokenizer, whether in classification tasks or vision & language tasks.
Regardless of model size, on most vision & language tasks, subword-level tokenizer tends to perform better than word-level tokenizer. The reason may be that subword-leved tokenizer retain a substantial amount of language-related information, although may not be highly relevant to visual information. This makes the features of the models trained with the subword-level tokenizer more readily integrated with large language models.
Overall, using a subword- level tokenizer exhibits better scaling behavior and is more suitable for use in large multi-modal models.
Different subword-level tokenizers Table 5 presents the results on classification tasks and LLaVA downstream tasks with different subword-level tokenizers. Here, we compare the character-based byte pair encoding tokenizer [64] used in CLIP [57], the WordPiece [72] tokenizer used in BERT [16] and the SentencePiece [36] tokenizer used in LLama [69], they are all subword-level tokenizers. The tokenizer used in openai CLIP obtains best performance on the classification task and LLaVA downstream tasks. Finally, we choose the tokenizer used in CLIP [57] for the training of SuperClass models.
Classification loss Table represents different classification loss on ImageNet- 1k dataset. We selected several of the most commonly used multi-label classification losses for experimentation. Softmax loss is often used in single-label classification tasks. possible apply a softmax loss in multi-label scenario through describing labels in probabilistic way. BCE loss is binary cross-entropy (BCE) loss and often used in multi-label classification tasks as baseline. Asymmetric Loss(ASL loss) improved BCE loss to address nositive-negative imbalance. Soft margin loss is

---

Table 6: The performance classification tasks with different classification losses. All models trained the same settings with batch size 16k, 512M seen samples and ViT-B/16 as Backbone.
Accs&Acc.
Softmax
BCE
ASL
SoftMargin
Two-way
Linear prob
75.6
73.6
73.8
73.5
74.8
Zero-shot
60.8
58.5
58.7
58.1
59.7
Table 7: The effect of IDF weight in the loss and removing stopwords.
Classification
Vision & Language Downstream Tasks
MMBench
VQAv2
VizWiz
T-VQA
MMMU
GQA
SQA
MME
POPE
SEED
Tokenizer
LP
ZS
SuperClass
76.0
61.7
68.34
57.43
41.79
24.41
65.54
54.03
1324
82.28
36.9
52.88
w/o IDF
75.6
60.8
68.08
57.27
47.60
23.73
65.44
54.55
1310
82.58
34.6
52.53
rm Stopwords
75.7
61.0
68.29
57.41
47.67
24.12
65.34
53.86
1343
82.50
33.8
53.12
margin-based based loss for multi-label classification tasks. Two-way loss is the current state-of-the-as (SOTA) for multi-label classification tasks.
Surprisingly, the simplest softmax loss outperforms all other multi-label classification losses by a large margin. We believe that existing multi-label classification losses operate under the assumption that labels are both accurate and complete, aiming to optimize the classification margin between positive and negative classes. However, in reality, image-text data contains considerable noise, and a single text passage cannot possibly capture all the contents an image. Consequently, certain categorical objects present in the image may not be mentioned in the associated text. In the context of image-text pretraining, how to design better loss function remains question worthy of exploration.
IDF as class weights Considering that the importance of each category (subword) in the vocabulary not equal and the information they carry varies, we use IDF as class weights. The Table repsents the results of with and without IDF as class weights. SuperClass without IDF experienced a noticeable decrease in accuracy on classification tasks, the change in precision on LLaVA tasks is not significant.
Removing stopwords? Stop words commonly used in Text Mining and Natural Language Processing (NLP) to eliminate words that are so widely used that they carry very little useful removed. The stopwords
information. In the previous classification methods, the stopwords download from NLTK [49]. However, the results in Table shows that the keeping stopwords could help the vision encoder gain better performance classification tasks.
5 Limitation and Conclusion
We have conducted thorough comparison of vision encoders pre-trained with contrastive and classification objectives and determined that models pre-trained with classification surpass CLIP models in both classification and vision & language tasks. Additionally, our approach does not require a text encoder, which leads to higher training efficiency than that of CLIP. Furthermore, our findings suggest that classification model and data sizes increase, and we encourage future research delve into this possibility.
pre-training task may have beneficial scaling properties as
While it delivers impressive results on various downstream tasks, it completely ignore word order and object relationships, which implies that we losing important supervisory information. Addressing this will be the direction of our future research efforts.
To up, have demonstrated that straightforward image classification serve effective pre- trainin strategy for vision backbones derived from image-text data. Our aim is to stimulate subsequent studies to pay more attention to the benefits of classification pre-training task for vision encoders.

---

References
[1] lean-Baptiste Alayrac, Jeff Donahue, Pauline Lue, Antoine Micch, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han. Zhitao Gong, Sima Samangooei. Marianne Monteiro, Jacob Menick, Sebastian Borgeaud, Andrew Brock, Aida Nematzadeh, Sahand Sharifzadeh, Mikolaj Binkowski, Ricardo Barreira, Oriol Vinyals, Andrew Zisserman, and Karen Simonyan. Flamingo: A visual language model for few-shot learning. In NeurIPS, 2022.
[2] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
[3] Hangbo Bao, Li Dong, Songhao Piao, and Furu Wei, Beit: Bert pre-training of image transformers, arXiv preprint Xiv: 082054 2021.
[4] Minwoo Byeon, Beomhee Park, Haccheon Kim, Sungjun Lee, Woonhyuk Back, and Sachoon Kim. Coyo-700m: Image-text pair dataset. httpa://gitiee www.kaaaaobarinneceooooaaaaaaaaa
2022.
[5] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in sule--up vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[6] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12M: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In CVPR, 2021.
[7] Jienneg Chen, Qihang Yu. Xiaohui Shen, Alan Yuille, and Liang- Liang-Chieh Chen. Vitamin: Designing scalable vision models in the vision-language era. arXiv preprint arXiv:2404.0213 021122 2024.
[8] Xiaokang Chen, Mingyu Ding, Xiaodi Wang. Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang. Context autoencoder for self self-suuer representation learning. International Journal of Computer Vision, 132(1):208 2233 2024.
[9] Xinlei Chen, Hao Fang, Tsung-Yr Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and
C. Lawrence Zitnick. Microsoft COCO Captions: Data collection and evaluation server. arXiv preprint arXiv:1504.0033: 2015.
[10] Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training sell-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9640-9649, 2021.
[11] Zhe Chen, Jiannan WU, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. Internvl: Scaling up vision foundation models and aligning for generic vinual-linguitti tasks. arXiv preprint arXiv:2312. 14238, 2023.
[12] Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 28199, 2023.
[13] Wei-L.in in Chiang, Zhuohan Li, 7i Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, et al. Vicuna: An open-source chathot impressing gpt-4 with 90% * chatept quality, march 2023. URL http:://lmsys. org/blog/2023-00 000-0030-3--30--m--e 3(5), 2023.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datahase. In 2009 IEEE conference on computer vision and pattern recognition, pages 248-255. Iece, 2009.
[15] Karan Desai and Justin Johnson. VirTex: Learning visual representations from textual annotations. In CVPR, 2021.
[16] Jacob Devlin, Ming-Wei Chang, Kenton Lec, and Kristina Toutanova. Bert: Pre-training of deep bidirec- tional transformers for language understanding. arXiv preprint arxin:1810.04800 2018.
[17] Xiaoyi Dong. Jianmin Bao, Ting Zhang, Dongdong Chen, Shuyang Gu, Weiming Zhang, Lu Yuan, Dong Chen, Fang Wen, and Nenghai Yu. Clip itself is a strong fine-tuner: Achieving 85.7% and 88.0% top-1 accuracy with vit-h and vit-I on imagenet. arXiv preprint arXiv:2212.066 2022.
[18] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11922 2020.
10

---

[19] Yuxin Fang, Wen Wang, Binhui Xie. Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang. and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 19358-19369, 2023.
[20] Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Zhenyu Qiu, Wei Lin. Jinrui Yang. Xiawu Zheng, et al. Mme: A comprehensive evaluation benchmark for multimodal large language models. arXiv preprint arXiv:2306. 13394, 2023.
[21] Samir Yitzhak Gadre, Gabriel Ilharco. Alex Fang. Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, al. Datacomp: In search of the next generation of multimodal datasets. arXiv preprint arXiv:2304. 14100, 2023.
[22] Yash Goyal, Tejas Khot, Douglas Summers Stay, Dhruv Batra, and Devi Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
[23] Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jicbo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3608 3617, 2018.
[24] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 16000-16009, 2022.
[25] Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song. and Jacob Steinhardt. Mcasuring massive multitask language understanding. arXiv preprint 2rXiv:2009003300, 2020.
[26] Xiaowei Hu, Zhe Gan, Jianfeng Wang. Zhengyuan Yang. Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning, In CVPR, pages 17980-17989, 2022.
[27] Xinyu Huang. Youcai Zhang, Jinyu Ma. Weiwei Tian, Rui Feng, Yuejie Zhang, Yaqian Li, Yandong Guo, and Lei Zhang. Tag2text: Guiding vision-language model via image tagging. arXiv preprint arXiv:2303.0566 2023.
[28] Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700-6700 6709,60990000 2019.
[29] Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongscok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021.
[30] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham. Quoe V. Le, Yun-Hsuan Sung. Zhen Li, and Tom Ducrig. Scaling up visual and rision-languae representation learning with noisy text supervision. In /CML, 2021.
[31] Armand Joulin, Laurens Van Der Maaten, Allan Jabri, and Nicolas Vasilache. Learning visual features from large weakly supervised data, In Computer Visi 2016: 14th Eumoean Conference, Amsterdam, The Netherlands, October I| 1-14, 2016, Proceedings, Part VII 14, pages 67-84. Springer, 2016
[32] Geewook Kim, Teakgyu Hong, Moonbin Yim, JeongYeon Nam, Jinyoung Park, Jinyeong Yim, Won- scok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. OCR-Free document understanding transformer. In ECCV, pages 498-517, 2022.
[33] Takumi Kobayashi. Two-way multi-label loss. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7476-7485, 2023.
[34] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages
2013.
[35] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[36] Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv: 1808. 06226, 2018.
[37] Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew Dai, Zhifeng Chen, et al. Mammut: A simple architecture for joint learning for multimodal tasks, arXiv:2303. 16839, 2023.

---

[38] Kenton Lee. Mandar Joshi, Iulia Ture, Hexiang Hu. Fangyu Liu, Julian Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wai Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing pretraining for visual language understanding, arXiv:2210.03347, 2022.
[39] Ang Li, Allan Jabri, Armand Joulin, and Laurens Van Der Maaten. Learning visual n-grams from web data, In ICCV, pages 4183-4192, 2017.
[40] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal Ilms with generative comprehension. arXiv preprint arXiv:2307.16125, 2023.
[41] Gang Li and Yang Li. Spotlight: Mobile UT understanding using vision-language models with a focus. In ICLR, 2023.
[42] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. BLIP: Bootstrapping language-image pre-training for unified vision- laiiun-a understanding and generation. In ICML, pages
2022.
[43] Wen Li, Limin Wang. Wei Li, Eirikur Agustsson, and Lue Van Gool. Webvision database: Visual learning and understanding from web data. arXiv preprint 1708. 02862, 2017.
[44] Xianhang Li. Zeyu Wang. and Cihang Xie. An inverse scaling law for clip training. Advances in Neural Information Procexsing Systems, 36, 2024.
[45] Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming He. Scaling language -inge pre-training via masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13300-23400, 2023.
[46] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji rong Wen. Evaluating object hallucination in large vision-language models. ArXiv, hts//2055.10355, 2023.
[47] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning, 2023.
[48] Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model all-around player? arXiv preprint Xix::300700628/ 2023.
[49] Edward Loper and Steven Bird. Nitk: The natural language toolkit. arXiv preprint cs/0205028, 2002.
[50] Timo Lüddeekke and Alexander Eeker. Image segmentation using text and image prompts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7086-7096, 2022.
[51] Sachin Mehta, Maxwell Horton, Fartash Faghri, Mohammad Hossein Sekhavat, Mahyar Najibi, Mehrdad Farajtabar, Oncel Tuzel, and Mohammad Rastegari. Catlip: Clip-level visual recognition accuracy with 2.7 faster pre- pre-training on web-scale image-text data. arXiv preprint arXiv:2404. 2et.:22444.1553,.. 2024.
[52] M Minderer, A Gritsenko, A Stone, M Neumann, D Weissenborn, A Dosovitskiy, A Mahendran, A Arna,, M Dehghani, Z Shen, al. Simple open -pen-vocabulary object detection with vision transfomeers arxiv 2022. arXiv preprint arXiv:2205.022 06230, 2, 2022.
[53] Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.0977 2021.
[54] Yasuhide Mori, Hironobu Takahashi, and Ryuichi Oka. Image-to-wee word transformation based on dividing and vector quantizing images with words. In First international workshop on multimedia intelligent storage and retrieval management, volume 2. Citeseer, 1999.
[55] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacddin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.0719 004220444779,, 2023.
[56] Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, and CV Jawahar. Cats and dogs. In 20/2 IEEE conference on computer vision and pattern recognition, pages 3498-3505. IEFE, 2012.
[57] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021.
[58] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models unsupervised multitask learners. 2019.
12

---

[59] Tal Ridnik, Emanuel Ben-Baruch, Nadav Zamir, Asaf Noy, Itamar Friedman. Matan Protter, and Lihi Zelnik- Manor. Asymmetric loss for multi-label classification. In Proceedings of the IEEE/CVF international conference on computer vision, pages 82-91, 2021.
[60] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[61] Mert Bulent Sariyildiz, Julien Perez, and Diane Larlus. Learning visual representations with caption annotations. In ECCV, pages 153-170, 2020.
[62] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 25294, 2022.
[63] Christoph Schuhmana, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki, Laion-400m: Open datasct of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:21 2021.
[64] Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. arXiv preprint Xrxiv: (000::508.000000 2015.
[65] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018
[66] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattem recognition, pages 8317-8326, 2019.
[67] Quan Sun. Qiying Yu. Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.052222 2023.
[68] Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCCI00M: the new data in multimedia research. Commun. ACM, 59(2):64-73, 2016.
[69] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 2023.
[70] Michael Tschannen, Manoj Kumar, Andreas Steiner, Xiaohua Zhai, Neil Houlsby, and Lucas Beyer. Image captioners are scalable vision learners too. Advances in Neural Information Processing Systems, 36, 2024.
[71] Zirui Wang, Jiahui Yu, Adams Wei Yu. Zihang Dai, Yulia Tsvetkov, and Yuan Cao. SimVLM: Simple visual language model pretraining with weak supervision. In ICLR, 2022.
[72] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoe V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao. Qin Gao, Klaus Macherey, et al. Google's s neural machine translation system: Bridging the gap between human and machine translation. arXiv preprini arXiv:1609.08/44 2016.
[73] Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9653-9663, 2022.
[74] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IERE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144 2022.
[75] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Trans. Machine Learning Research, 2022.
[76] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training, arXiv:2303.1534. 2023.
[77] Xiaohua Zhai, Xiao Wang. Basil Mustafa, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18123 18133, 2022.
13

---

[78] Youcai Zhang, Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xie. Yuzhuo Qin. Tong Luo, Yaqian Li, Shilong Liu, et al. Recognize anything: A strong image tagging model. arXiv preprint urXiv:2306.0333 033114 2023.
[79] Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal chain-of-thoughi reasoning in language models. arXiv preprint arXiv:2302.0092, 2023.
[80] Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre- pre-training with online tokenizer. International Conference on Learning Representations (ICLR), 2022.
14

---

Table 8: The performance of classification and LLaVA downstream tasks with different samples. "LP" means linear probing and "ZS" means zero-shot classification, these two are tested on ImageNet- 1K dataset. The results of vision&language downstream tasks are obtained by combine the frozen vision models and Vicuna-7B [13], following the settings in LLaVA [47].
Classification
Vision & Language Downstream Tasks
MMBench
VQAv2
VizWiz
T-VQA
MMMU
GQA
SQA
MME
POPE
SEED
Method
Data
LP
ZS
CLIP
128M
69.4
48.8
63.20
53.98
45,80
13.27
64.70
46.56
1216
78.20
35.1
47.84
SuperClass
128M
71.1
51.2
64.07
54.96
49.21
13.33
65.44
49.14
1241
80.01
35.3
49.08
CLIP
512M
79.7
65.9
68.13
57.32
44,92
22.21
64,45
51.54
1299
82.48
35.3
53.06
SuperClass
512M
80.5
69.0
70.40
58.16
51.48
29.83
67.72
59.45
1373
84.04
36.3
53.74
CLIP
1.28B
81.9
71.4
70.33
58.95
46.71
27.97
64.65
55.49
1351
83.37
35.7
55509
SuperClass
1.28B
82.6
73.6
71.85
59.09
51.70
34.37
65.94
59.70
1392
84.41
36.8
55.51
Table 9: The performance of classification and LLaVA downstream tasks with different model sizes.
Classification
Vision & Language Downstream Tasks
MMBench
VQAv2
VizWiz
T-VQA
MMMU
GQA
SQA
MME
POPE
SEED
Method
VIT
LP
ZS
CLIP
S/16
63.6
52.0
64.48
55.26
44.16
15.98
65.84
47.33
1227
80.49
35.8
49.72
SuperClass
S/16
67.9
52.8
65.54
55.95
46.23
16.46
65.64
48.53
1306
81.02
33.2
50.43
CLIP
B/16
75.5
59.8
66.11
56.28
49.17
19. 10
64.30
48.62
1289
81.47
35.9
50.76
SuperClass
B/16
76.0
61.7
68.34
57.43
41.79
24.41
65.54
54.03
1324
82.28
36.9
52.88
CLIP
I/16
79.7
65.9
68.13
1.
57.32
44.92
22.21
64.45
51.54
1299
82.48
35.3
53.06
SuperClass
L/16
80.5
69.0
70.40
58.16
51.48
29.83
67.72
59.45
1373
84.04
36.3
53.74
A
Appendix / supplemmenal material
Broader Impacts
This work presents new approach to train vision models, which can be applied for image recognition and other vision tasks. The approach demonstrates higher efficiency than the popular one in the community, which can reduce the computational cost and the power cost for computer vision model training.
Data Scaling results In Table 8, we showcase the performance across classification and vision & language tasks for varying seen samples. For a fair comparison, both CLIP and SuperClass models undergo training with identical settings, which include a batch size of 16k and ViT-L/16 as backbone.
Figure 2 illustrates that the number of seen samples grows, there is a noticeable improvement in performance for both classification and downstream tasks linked to LLaVA. Typically, models pre-trained with SuperClass outperform those trained with CLIP in terms of accuracy when given the same amount of seen samples. SuperClass exhibits the same or slightly better scaling behavior compared to CLIP on downstream tasks. addition, SuperClass does not require text encoder, offers better efficiency in training compared to CLIP.
Model scaling results Table 9, we showcase the performance across classification and vision language tasks for varying model scales. For a fair comparison, both CLIP and SuperClass models undergo training with identical settings, which include a batch size of 16k and 512 million seen samples.
15

---

Table 10: Performance of frozen visual representations trained via image classification (SuperClass) and constrastively (CLIP). Linear probing and zero-shot classification are both tested on ImageNet- Ik dataset. Captioning is conducted on COCO captions and CIDEr is reported in the table. The zero-shot accuracy of SuperClass is obtained after lock-image tuning [77].
Method
Backbone
Data
Seen Samples
Zero-shot
Linear Probing
CLIP
RN-50
Datacomp-1 IB
1.28B
60.73
70.28
SuperClass
RN-50
Datacomp- IB
1.28B
62.81
71.92
CLIP
Conv-Next-tiny
Datacomp-11
1.28B
59.94
70.35
SuperClass
Conv-Next-tiny
Datacomp-11
1.28B
62.85
72.33
Table 11: The performance of vision & language downstream tasks with different pretrained models.
Method
VQAv2
GQA
VizWiz
T-VQA
SciQA
MME
MMB
PoPE
MMMU
OpenCLIP
74.54
61.03
50.47
38.16
67.33
1434/269
60.73
85.52
35.9
MAE
63.50
54.58
50.22
11.55
54.75
1175/343
42.44
80.69
35.7
DINOv2
73.32
61.87
49.15
14.08
64.90
1336/297
57.90
86.24
35.3
SuperClass
75.24
60.96
54.33
39.20
66.09
1371/322
63.14
85.69
36.0
As shown in Figure 2, with the model scaling up, we observe a corresponding enhancement in performance, whether it is for classification tasks or the downstream tasks associated with LLaVA. Generally speaking, with the same model size, models pre-trained using SuperClass exhibit superior precision compared to those trained with CLIP. SuperClass demonstrates better scaling on zero-shot classification and VQAv2, T-VQA tasks.
Superclass with different model types To evaluate the robustness our proposed method across different model types, we selected two representative convolution-based networks: ResNet50 and ConvNext-Tiny, We compare SuperClass against CLIP for ImageNet zero-shot (LiT) and linear probing classification, shown in Table 10. All experiments were conducted with batch size of 16k and 1.28B seen samples. We observe that SuperClass surpasses CLIP in all settings by a clear margin, ranging from 1.64 to 2.91. These results demonstrate that the superiority of SuperClass over CLIP is robust across different model architectures.
VLM downstream tasks with different types of pretraining models Following the LLLVA setup, combine frozen CLIP models, self-supervised models, and SuperClass models with the pre-trained Vicuna-7B and perform downstream tasks. The experimental results in Table 11 demonstrate that the proposed method could achieve better than self-supervised ViT pre-training methods, like DINOv2, and weakly-supervised methods, like CLIP.
Comparison with other classification based pretraining models We have included the compari- son with other classification-basee methods, like CatLIP [51] in the subsection Word-level tokenizer vs. Subword-leved tokenizer. The word-level tokenizer is used in CatLIP [51], which carefully selected approximately 40,000 "gold labels" from the datacomp-IB dataset. Aside from the tokenizer being different, all models are trained under the same settings. The results of Table 4 show that with the increasing size of the model, the subword-level tokenizer gradually outperforms the word-level tokenizer, whether in classification tasks or vision & language tasks. We also provide the results of finetuning on ImageNet- 1k in Table 12. Using the same dataset Datacom- 1B for training, the same backbone ViT-L/16 as backbone, the same number 13 Billion of training samples seen, the SuperClass could achieve better performance than CatLII (87.8 vs 86.5).
Table 12: Comparison of the Fine-tuning top-1 accuracy ImageNet- 1K dataset. *number from the paper.
Method
Pretraining Data
ImageNet- Ik Fine-tuning
OpenCLIP VIT-L/14
Datacomp- 1B
87.4
CatLIP ViT-I/16*
Datacomp- IB
86.5
Superclass VYT-I/116
Datacomp- IB
87.8
16

---

1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: refer to abstract and introduction
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Discussed the conclusion
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification:
Guidelines:
17

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
Important details are provided in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g.. a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
18

---

Justification:
We use public data for model training. The code will be cleaned and made public in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurlPS code and data submission guidelines (https: //nip. cttpssccccppbccccccecccccccc guwd/g. /eodSS/a ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: we provide all the details in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: We follow the convention in prior works and report the performance number on the standard benchmarks.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
19

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: please refer to the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: https///////ee.eeeeeeeeee wttppps.cccpb.ccecii.... [ttppiic//iiccccuuueellllll csQuidel ines?
Question:
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss in the conclusion.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no socictal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We used public datasets.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
22


Reviews:
[{"review_id": "jpCU3uXdu1", "reviewer": "Reviewer_aefc", "summary": "This paper introduces a multi-label classification pre-training style for visual image encoder pre-training.", "questions": "- 1. The greatest advantage of CLIP is its zero-shot capacity across various downstream tasks. The zero-shot ability of SuperClass is not well-demonstrated on downstream tasks like zero-shot text-image retrieval, zero-shot text-video retrieval, and zero-shot STR. The reviewer thinks the authors should include an analysis of the zero-shot ability like the original CLIP paper.\n\n- 2. The baselines should not be CLIP-style pre-trained vision language models. The proposed method aims to pre-train a vision transformer. The baseline should be other weakly-supervised methods or self-supervised methods like MAE, etc. The authors should compare these methods in terms of training efficiency and performance of transfer learning. These methods also show good transfer learning ability, for example, MAE has good performance of transfer learning on COCO, while it is only pre-trained on ImageNet-1K.\n\n- 3. Some arguments about previous so-called \"bag-of-word classification\" pre-trained methods may not be correct. In the introduction, the authors claim \" However, these methods fail to gain popularity from the community, as most of the experiments are conducted on a small scale and there is no evidence showing their scalability to data size and model size in comparison to CLIP \". Nevertheless, CatCLIP[1] already conducts experiments with CLIP-H on DataComp-1.3B.\n    - 3.1. The authors should also compare with these \"bag-of-word classification\" pre-trained methods.\n\n- 4. The authors say \"All experiments were carried out on an A100 GPU equipped with 80GB of memory.\" Could the authors provide training times for the experiments in Table 1? Besides, are experiments in Table 2 also trained on an A100 80G? If so, could the authors provide the training time? \n\n[1] https://arxiv.org/pdf/2404.15653", "limitations": "Yes.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The proposed method is straightforward.", "weaknesses": "- The zero-shot capacity of such a multi-label pre-trained model is not well demonstrated.\n- The paper lacks a comprehensive comparision with weakly-supervised or unsupervised visual encoder methods. Only comparing with CLIP is not enough.\n- Some arguments about previous so-called \"bag-of-word classification\" pre-trained methods may not be correct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ">**Q1** Zero-shot capacity.\n\n**A1.** We thank the reviewer for triggering the discussion on zero-shot abilities of CLIP and our model. Honestly, our model does not come with trivial zero-shot image-text retrieval usage. However, we can enable this behavior by levering LiT [a], which learns a text encoder by keeping the image encoder fixed. As shown by the following table, we test the zero-shot ability of our model on 10 datasets. Superclass beats openCLIP on seven of the datasets. It is worth noting that SuperClass uses a ViT large model with patch size 16, while the baseline method adopts a patch size of 14. This makes the superiority of SuperClass over openCLIP even clearer.\n\n| Case       | Model    | Data        | Seen samples | IN-1K | imagenet-v2 | imagenet-r | imagenet-a | imagenet-sketch | GTSRB | Rendered SST2 | ObjectNet | SUN397 | Country211 |\n|------------|----------|-------------|--------------|-------|-------------|------------|------------|-----------------|-------|---------------|-----------|--------|------------|\n| openCLIP   | ViT-L/14 | Datacomp-1B | 12.8B        | 79.2  | 72.1        | 90.8       | 69.6       | 68.0            | 58.5  | 61.0          | 74.3      | 74.3   | 31.6       |\n| SuperClass | ViT-L/16 | Datacomp-1B | 12.8B        | 79.7  | 72.4        | 91.6       | 68.8       | 70.6            | 58.5  | 61.6          | 73.9      | 73.8   | 32.3       |\n\nMoreover, we'd like to emphasize that there is another increasingly more important application of the CLIP model. Specifically, CLIP is predominantly the default vision encoder in existing vision-language models (e.g., LLaVA, BLIP). We show that when combined with a large language model, SuperClass is able to substantially improve performance over CLIP. Please refer to Table 2 and Table 3 on the paper for details. \n\n[a] Zhai, Xiaohua, et al. \"Lit: Zero-shot transfer with locked-image text tuning.\" CVPR. 2022.\n\n>**Q2** The baseline should be other weakly-supervised methods or self-supervised methods like MAE, etc\n\n**A2.** Thanks for the suggestion. We have added more SOTA methods for comparison, including self-supervised methods (MoCov3, Dinov1&v2, MAE, BEiT, CAE) and weakly-supervised methods (CLIP, Cappa). Additionally, we have evaluated more downstream tasks and datasets, such as semantic segmentation on ADE20K and instance segmentation on COCO. \nFinally,  Following the LLaVA setup, we combine frozen CLIP models, self-supervised models, and SuperClass\nmodels with the pre-trained Vicuna-V1.5-7B and perform downstream tasks. \nThe experimental results demonstrate that the proposed method could achieve better than self-supervised ViT pre-training methods, like Dinov2, and weakly-supervised methods, like CLIP.\n\n**Linear probing on ImageNet-1K**\n\n| Method | Pre-training Data | ViT-Base | ViT-Base  | ViT-Large | ViT-Large |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|  |  | #Seen Samples | Top-1 (%) | #Seen Samples | Top-1 (%) |\n| contrastive or clustering based |  |  |  |  |  |\n| MoCov3 |  IN1K | 400M | 76.7 | 400M | 77.6 |\n| DINO | IN1K | 512M | 78.2 | - | - |\n| iBoT | IN22K | 400M | 79.5 | 256M | 81.0 |\n| DINOv2 | LVD-142M | 1.28B | - | 1.92B | 84.5 |\n| reconstruction based |  |  |  |  |  |\n| BEiT | D250M+IN22K | 1B | 56.7 | 1B | 73.5 |\n| SimMIM | IN1K | 1B | 56.7 | - | - |\n| CAE | D250M | 2B | 70.4 | 2B | 78.1 |\n| MAE | IN1K | 2B | 68.0 | 2B | 75.8 |\n| language-image pretraining based |  |  |  |  |  |\n| CLIP | WIT400M | 12.8B | 78.5 | 12.8B | 82.7 |\n| Cappa | WebLI-1B | - | - | 9B | 83.0 |\n| OpenCLIP | Datacomp-1B | - | - | 12.8B | 83.9 |\n| Superclass | Datacomp-1B | 12.8B | 80.2 | 12.8B | 85.0 |\n| Superclass | Datacomp-1B | 1.28B | 78.7 | 1.28B | 82.6 |\n| Superclass | Datacomp-1B | 512M | 75.6 | 512M | 80.5 |\n\n**Instance segmentation and semantic segmentation**\n\nResults of instance segmentation are obtained by using Mask R-CNN on COCO with an input resolution of 1024×1024. Semantic segmentation results are obtained by using UperNet on ADE20K with an input resolution of 512×512.\n| Method | #Seen Samples | Semantic Segmentation mIoU | Instance Segmentation APmask |\n|:---:|:---:|:---:|:---:|\n| Supervised | - | 49.9 | 43.9 |\n| MoCov3 | 400M | 49.1 | 44.0 |\n| BEiT | 400M | 53.3 | 47.1 |\n| CAE | 2B | 54.7 | 47.6 |\n| MAE | 2B | 53.6 | 47.2 |\n| CLIP | 12.8B | 57.9 | 48.3 |\n| Superclass | 1.28B | 56.2 | 48.1 |\n| Superclass | 12.8B | 58.4 | 49.0 |\n\n\n**VLM downstream tasks** \n| Model | Size | VQAv2 | GQA | VizWiz | TextVQA | SciQA | MME | MMBench | PoPE | MMMU |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| openCLIP | ViT-Large | 74.54 | 61.03 | 50.47 | 38.16 | 67.33 | 1434.86/268.92 | 60.73 | 85.52 | 35.9 |\n| MAE | ViT-Large | 63.5 | 54.58 | 50.22 | 11.55 | 64.75 | 1175.04/343.92 | 42.44 | 80.69 | 35.7 |\n| DINOv2 | ViT-Large | 73.32 | 61.87 | 49.15 | 14.08 | 64.9 | 1335.61/296.78 | 57.9 | 86.24 | 35.3 |\n| Superclass | ViT-Large | 75.24 | 60.96 | 54.33 | 39.2 | 66.09 | 1371.32/321.78 | 63.14 | 85.69 | 36 |\nopenCLIP and Superclass are trained with 12.8B seen samples.\n\n>**Q3** Some arguments about previous so-called \"bag-of-word classification\"\n\n**A3.** Thanks for pointing it out. We will rewrite the description here. CatLIP [46] is a concurrent work. We have already cited and compared it in the paper.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "PPpoDUbb0D", "reviewer": "Reviewer_W9bX", "summary": "This paper explores a new direction to pretrain vision backbones using large scale image-text pairs for learning visual representations which are suitable to various downstream tasks. More specifically, this work proposes a classification based objective function as an effective alternative to CLIP's standard cross-modality similarity based constrastive loss. The proposed model SuperClass uses a image encoder to map image to token probabilities having a head size equal to CLIP's tokenizer vocabulary. The texts are converted into labels using weighted tokenizer. Both CLIP baseline and SuperClass are pretrained on datacomp image-text pairs dataset and evaluation results across various vision and vision-language benchmarks are reported. The proposed method shows greater efficiency due to being text-encoder free and also performs favorably well over previous approaches. Extensive ablation studies are performed to justify the design choices made in the paper.", "questions": "Please refer to the weaknesses section for the questions. I will highly recommend the authors submit a rebuttal response. I will be happy to reconsider my final scores based on the response from the authors.", "limitations": "Yes the authors have adequately addressed the limitations in the paper.", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "**Strengths:**\n1) The idea of pretraining large scale vision models using classification objectives is very motivating, as it provides advantages over image-text contrastive loss such as compute efficiency, disentangling the role of text embeddings etc. \n2) The proposed method is simple and effective.\n3) The experimental results are favorable for SuperClass against its direct baseline CLIP.\n4) The choice of the components in the proposed method such as IDF based weighting, loss function and tokenization has been validated in the paper via ablation studies.\n5) Paper is easy to read and understand.", "weaknesses": "**Weaknesses**\n1) In my understanding, one of the weaknesses of this work is lack of comparisons with related works. For example, the only method with which SuperClass is compared with is CLIP which is only a baseline. I believe there should be comparisons with other related SOTA works. \n2) The proposed approach might not be capable of doing multi-label classification or zero-shot classification as compared to other competitors such as RAM, CLIP etc. Also CLIP has out of the box additional features such as prompt ensembling, zero-shot segmentation [1] etc which might be not present in SuperClass. This would question how it can comprehensively show effectiveness over CLIP and other vision-language models.\n3) It is not clear how the proposed method learns suitable representations when using the subwords as labels which does not correspond to any visual concept. I believe there is a bit of analysis missing in the manuscript.\n4) The results for the zeroshot and linear probing in Tab1 and Tab 2 of the main paper are different for the same model. It is unclear if why the results are different for same model.\n\n\n[1] Extract Free Dense Labels from CLIP (ECCV 2022)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1**: Comparisons with other related SOTA works\n\n**A1**. Thanks for the advice. We have added more SOTA methods for comparison, including self-supervised learning methods (MoCov3, Dinov1&v2, MAE, BEiT, CAE) and weakly-supervised methods (CatLIP, Cappa). Additionally, we have evaluated more downstream tasks and datasets, such as semantic segmentation on ADE20K and instance segmentation on COCO. Please refer to our response to Reviewer1(aefc) and Reviewer4(bBvT) for more results.\n\n\n**Linear probing on ImageNet-1K**\n\n| Method | Pre-training Data | ViT-Base | ViT-Base  | ViT-Large | ViT-Large |\n|:---:|:---:|:---:|:---:|:---:|:---:|\n|  |  | #Seen Samples | Top-1 (%) | #Seen Samples | Top-1 (%) |\n| contrastive or clustering based |  |  |  |  |  |\n| MoCov3 |  IN1K | 400M | 76.7 | 400M | 77.6 |\n| DINO | IN1K | 512M | 78.2 | - | - |\n| iBoT | IN22K | 400M | 79.5 | 256M | 81.0 |\n| DINOv2 | LVD-142M | 1.28B | - | 1.92B | 84.5 |\n| reconstruction based |  |  |  |  |  |\n| BEiT | D250M+IN22K | 1B | 56.7 | 1B | 73.5 |\n| SimMIM | IN1K | 1B | 56.7 | - | - |\n| CAE | D250M | 2B | 70.4 | 2B | 78.1 |\n| MAE | IN1K | 2B | 68.0 | 2B | 75.8 |\n| language-image pretraining based |  |  |  |  |  |\n| CLIP | WIT400M | 12.8B | 78.5 | 12.8B | 82.7 |\n| Cappa | WebLI-1B | - | - | 9B | 83.0 |\n| OpenCLIP | Datacomp-1B | - | - | 12.8B | 83.9 |\n| Superclass | Datacomp-1B | 12.8B | 80.2 | 12.8B | 85.0 |\n| Superclass | Datacomp-1B | 1.28B | 78.7 | 1.28B | 82.6 |\n| Superclass | Datacomp-1B | 512M | 75.6 | 512M | 80.5 |\n\n**Instance segmentation and semantic segmentation**\n\nResults of instance segmentation are obtained by using Mask R-CNN on COCO with an input resolution of 1024×1024. Semantic segmentation results are obtained by using UperNet on ADE20K with an input resolution of 512×512.\n| Method | #Seen Samples | Semantic Segmentation mIoU | Instance Segmentation APmask |\n|:---:|:---:|:---:|:---:|\n| Supervised | - | 49.9 | 43.9 |\n| MoCov3 | 400M | 49.1 | 44.0 |\n| BEiT | 400M | 53.3 | 47.1 |\n| CAE | 2B | 54.7 | 47.6 |\n| MAE | 2B | 53.6 | 47.2 |\n| CLIP | 12.8B | 57.9 | 48.3 |\n| Superclass | 1.28B | 56.2 | 48.1 |\n| Superclass | 12.8B | 58.4 | 49.0 |\n\n\n**VLM downstream tasks** \n| Model | Size | VQAv2 | GQA | VizWiz | TextVQA | SciQA | MME | MMBench | PoPE | MMMU |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| openCLIP | ViT-Large | 74.54 | 61.03 | 50.47 | 38.16 | 67.33 | 1434.86/268.92 | 60.73 | 85.52 | 35.9 |\n| MAE | ViT-Large | 63.5 | 54.58 | 50.22 | 11.55 | 64.75 | 1175.04/343.92 | 42.44 | 80.69 | 35.7 |\n| DINOv2 | ViT-Large | 73.32 | 61.87 | 49.15 | 14.08 | 64.9 | 1335.61/296.78 | 57.9 | 86.24 | 35.3 |\n| Superclass | ViT-Large | 75.24 | 60.96 | 54.33 | 39.2 | 66.09 | 1371.32/321.78 | 63.14 | 85.69 | 36 |\nopenCLIP and Superclass are trained with 12.8B seen samples.\n\n\n> **W2**: The proposed approach might not be capable of doing multi-label classification or zero-shot classification ...how it can comprehensively show effectiveness over CLIP and other vision-language models.\n\n**A2**. Our paper focuses on the issue of vision encoder pretraining, specifically whether the pre-trained model can perform better in vision-language models. Therefore, the native zero-shot capability may not be the primary property we consider.\nHowever, we can use Lock Image Tuning(LiT)[69] to equip our pre-trained vision backbone with zero-shot classification and retrieval capabilities. After LiT, we also could do zero-shot segmentation using MaskCLIP[a]. The experimental results show that SuperClass could achieve much better performance on the Pascal context and COCO stuff dataset.\n| Method     | Backbone | #Seen sample | PASCAL Context | COCO Stuff |\n|------------|----------|--------------|----------------|------------|\n| CLIP       | ViT-B/16 | 1.28B        | 16.2           | 8.7        |\n| Superclass | ViT-B/16 | 1.28B        | 20.2(+4.0)     | 13.2(+4.5) |\n\n[a] Extract Free Dense Labels from CLIP (ECCV 2022)\n\n\n\n> **W3**:  using the subwords as labels does not correspond to any visual concept\n\n**A3**. Firstly, it is important to note that a single object class can be mapped to one or multiple subwords. Here is a detailed explanation of how our method works in both scenarios:\n1. Single Subword Mapping:\n  - When a class is mapped to a single subword, the process is similar to traditional classification tasks. The model learns to associate the subword with the corresponding visual patterns.\n  - This is akin to standard classification where each class is represented by a unique label, and the model learns the association between the label (subword) and the visual features.\n2. Multiple Subword Mapping:\n  - When a class is mapped to multiple subwords, our optimization objective is to maximize the co-occurrence probability of subwords that belong to the same class.\n  - This means the model learns to associate multiple subwords with the corresponding visual patterns, effectively capturing the relationship between subwords and the visual concept they represent.\n\n\n\n> **W4**: The results for the zero-shot and linear probing in Tab1 and Tab 2 of the main paper are different for the same model.\n\n**A4**. Because the models are trained with different #seen samples. In Table 1 and Table 2, we use the ViT-L/16 as the backbone. The models in Table 1 are trained with 12.8B seen samples. In Table 2, we study the effect of different seen samples. The models are trained with 128M, 512M, and 1.28B seen samples, respectively.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8QkWXfQcDY", "reviewer": "Reviewer_3a7k", "summary": "This paper proposes a simple classification-based vision-language pretraining method. The proposed SuperClass approach directly uses an off-the-shelf subword-level tokenizer to obtain the classification labels from raw text, without requiring any preprocessing. Then, the vision encoder is trained by optimizing the multi-label softmax loss, with Inverse Document Frequency (IDF) as the weight of each label. SuperClass achieves promising performance on classification and various vision-language tasks. Ablation experiments are conducted to validate the impact of different design choices.", "questions": "Please refer to the weaknesses.", "limitations": "The authors discuss some of the limitations of their work in Section 5. But I would like them to consider some of my concerns above.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The proposed method in this paper reveals the potential of classification in vision-language pretraining, which provides empirical evidence for further researching.\n2. With a simple framework that requires no data preprocessing, SuperClass enables large-scale training on paired image-text data. Compared to previous classification-based pretraining methods, the proposed approach demonstrates greater practical applicability.\n3. SuperClass is training-efficient by removing the need for a text encoder, and the extensive experimental results demonstrate the effectiveness and scalability of the proposed approach.", "weaknesses": "1. The robustness of the proposed method to different model types remains unclear. All the experiments in this paper use ViT as the vision encoder, and there is no evidence to demonstrate the effectiveness of the SuperClass on other encoder architectures, such as ResNet.\n2. The ablation experiment on different classification losses is only conducted on classification tasks, using a ViT-B/16 backbone and 512M seen data samples. It is important to demonstrate the robustness of the softmax loss on a broader range of vision-language tasks, beyond just classification. Furthermore, the impact of the choice of loss function on the scalability of the proposed method is not discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1**: The robustness of the proposed method to different model types remains unclear. \n\n**A1.** Thank you for the valuable advice. To evaluate the robustness of our proposed method across different model types, we selected two representative convolution-based networks: ResNet50 and ConvNext-Tiny. We compare SuperClass against CLIP for ImageNet zero-shot (LiT) and linear probing classification, as shown in the table below. All experiments were conducted with a batch size of 16k and 1.28B seen samples.\n\n| Method     | Backbone      | Zero-shot | Linear Probing |\n|------------|---------------|-----------|----------------|\n| CLIP       | RN-50         | 60.73     | 70.28          |\n| Superclass | RN-50         | 62.81     | 71.92          |\n| CLIP       | ConvNext-tiny | 59.94     | 70.35          |\n| Superclass | ConvNext-tiny | 62.85     | 72.33          |\n\nWe observe that SuperClass surpasses CLIP in all settings by a clear margin, ranging from 1.64 to 2.91. These results demonstrate that the superiority of SuperClass over CLIP is robust across different model architectures. \n\n> **W2**. The robustness of softmax loss on a broader range of vision-language tasks\n\n**A2**. Thank you for the advice. We further compare the sofrmax loss against different losses on various vision-language tasks. We selected three model sizes (ViT-S/16, ViT-B/16 and ViT-L/16) and trained them on Datacomp-1b with 512 million seen samples. The detailed results are presented in the following table. We observe that softmax is the best-performing loss function across different vision-language tasks and model sizes. \n\n|   Loss  |   dataset   | Seen Sample | Backbone | IN 0-shot | Linear Prob | VQAv2 (val) |  GQA  | VizWiz (val) | TextVQA (val) | SciQA (Img) | MMBench (en/cn) |    MME (P/C)   |  POPE |  MMMU | SEEDBench |\n|:-------:|:-----------:|:-----------:|:--------:|:---------:|:-----------:|:-----------:|:-----:|:------------:|:-------------:|:-----------:|:---------------:|:--------------:|:-----:|:-----:|:---------:|\n| Twoway  | Datacomp-1b | 512M        | vit-S/16 | 50.6      | 66.3        |    65.44    | 55.65 |     43.5     |     15.13     |    66.44    |   51.71/40.72   | 1278.42/335.35 | 79.81 | 34.41 |   50.04   |\n| BCE     | Datacomp-1b | 512M        | vit-S/16 | 47.7      | 65.8        |    64.49    | 55.63 |     48.15    |     13.27     |    65.99    |   50.08/39.86   | 1282.30/323.92 | 79.87 |  35.3 |   50.12   |\n| ASL     | Datacomp-1b | 512M        | vit-S/16 | 48.3      | 66          |    64.58    | 55.49 |     44.96    |     13.44     |    65.59    |   50.77/40.80   | 1290.92/315.15 | 80.46 |  35.1 |   49.98   |\n| Softmax     | Datacomp-1b | 512M        | vit-S/16 | 51.7      | 67          |     65.6    | 56.03 |     43.29    |     16.61     |    64.65    |   49.91/41.92   | 1315.08/306.78 | 81.46 |  35.8 |   51.03   |\n| Twoway  | Datacomp-1b | 512M        | vit-B/16 | 59.7      | 74.8        |    68.05    | 57.79 |     47.35    |     22.09     |    66.63    |   54.55/46.04   | 1350.92/335.71 | 82.51 |  36.8 |   53.02   |\n|  Margin | Datacomp-1b | 512M        | vit-B/16 | 58.1      | 73.5        |    67.08    | 56.67 |     44.17    |     17.87     |     64.9    |   53.43/42.95   | 1341.75/312.14 | 81.71 |  34.7 |   52.32   |\n| BCE     | Datacomp-1b | 512M        | vit-B/16 | 58.5      | 73.6        |    67.35    | 56.91 |     49.2     |     18.34     |    64.95    |   52.49/43.47   | 1327.92/332.14 | 81.89 |  36.7 |   52.76   |\n| ASL     | Datacomp-1b | 512M        | vit-B/16 | 58.7      | 73.8        |    67.59    | 57.02 |     47.02    |     19.01     |    65.44    |   54.72/46.39   |  1345.54/357.5 | 81.93 |  35.3 |   52.75   |\n| Softmax     | Datacomp-1b | 512M        | vit-B/16 | 60.8      | 75.6        |    68.08    | 57.27 |     47.6     |     23.73     |    65.44    |   54.55/46.13   | 1310.65/335.00 | 82.58 |  34.6 |   52.53   |\n| Twoway  | Datacomp-1b | 512M        | vit-L/16 | 66.7      | 78.3        |     70.2    | 58.36 |     46.25    |     27.21     |    64.35    |   57.30/48.62   | 1365.98/315.00 | 82.97 |   36  |   53.87   |\n| BCE     | Datacomp-1b | 512M        | vit-L/16 | 64.9      | 77.2        |    69.56    | 57.93 |     48.62    |      24.9     |     64.3    |   57.64/47.33   | 1316.55/355.71 | 83.17 |  35.1 |   54.01   |\n| ASL     | Datacomp-1b | 512M        | vit-L/16 | 66.1      | 77.6        |    69.71    | 58.43 |     51.16    |     25.49     |    65.49    |   58.41/49.48   | 1389.29/330.35 | 83.51 |  34.6 |   54.07   |\n| Softmax     | Datacomp-1b | 512M        | vit-L/16 | 68.3      | 80.1        |    70.27    | 58.03 |     48.98    |     28.87     |    67.03    |   57.30/49.14   | 1334.33/366.42 | 83.36 |  35.4 |   54.41   |\n\n> **W3**. The scaling properties of loss function\n\n**A3**. Due to the limited time for rebuttal, we only ablated the effect of different losses with different model sizes. As the model size increases, softmax consistently achieves the best accuracy while also demonstrating equal or better scalability compared to other losses.\n\n|         | ViT-S/16 | ViT-S/16 | ViT-B/16 | ViT-B/16 | ViT-L/16 | ViT-L/16 |\n|:-------:|:--------:|:--------:|:--------:|:--------:|:--------:|:--------:|\n|         |    ZS    |    LP    |    ZS    |    LP    |    ZS    |    LP    |\n|  Twoway |   50.6   |   66.3   |   59.7   |   74.8   |   66.7   |   78.3   |\n|  Margin |   47.4   |   65.6   |   58.1   |   73.5   |   64.6   |   76.9   |\n|   BCE   |   47.7   |   65.8   |   58.5   |   73.6   |   64.9   |   77.2   |\n|   ASL   |   48.3   |   66.0   |   58.7   |   73.8   |   66.1   |   77.6   |\n| Softmax |   51.7   |   67.0   |   60.8   |   75.6   |   68.3   |   80.1   |", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cbIk5xaAos", "reviewer": "Reviewer_bBvT", "summary": "The paper proposes a simple alternative to CLIP-style pretraining that doesn't require a text encoder and can be done using only a text tokenizer. The goal is to provide a simpler yet more efficient alternative to vision-language model (VLM) pretraining, which is known to be very expensive. Additionally, the authors simplify the VLM pertaining setup into a classification task which is both novel and very intuitive. \nThe authors demonstrate the efficiency of their method through experiments on both classification and vision-and-language downstream tasks. Additionally, they present a comprehensive set of ablations to dissect the performance gains of their proposed approach.\n\nWhile there are some limitations to their proposed method, the simplification offers valuable benefits for traditional classification and vision-language tasks.", "questions": "1. In Table 7, the authors show that removing stop-words has no effect and state in line 279 that \"keeping stopwords could help the vision encoder.\" This statement is not explained and seems counter-intuitive. Could the authors elaborate on this observation?\n2. The authors claim they can use IDF weights in an online manner. Could the authors explain how they achieve this? I don't understand how this process can be done online.\n3. The authors demonstrate that the performance of SuperClass is as good as or better than traditional CLIP-style pretraining on ImageNet-1K classification. However, it would be interesting if the authors could specify which classes benefit more from their method compared to traditional CLIP. Such an analysis could help understand where the gains stem from.", "limitations": "1. The authors acknowledge that their approach completely ignores word order, which can significantly impact the encoder's ability to understand tasks requiring spatial reasoning.\n2. Additionally, this approach may not be suitable for text-to-image systems, as these models need to understand word order and infer relationships from text to generate accurate images.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well-written and easy to follow, with clear motivation and well-planned experiments.\n2. The proposed method simplifies vision-language pretraining by eliminating the need for a text encoder, using a text tokenizer instead. This approach demonstrates comparable or better performance on both classification and vision-language tasks. It effectively transforms the contrastive pretraining task into an open vocabulary classification task, where text tokens provide supervision. This insight is powerful as it can help the community develop strong vision encoders more efficiently.\n3. The experiments and ablations presented are thorough. The authors systematically analyze every component of their method, including data, model scale, and the tokenizer.\n4. The paper also highlights that for simple classification tasks or vision question-answering tasks, CLIP-like pretraining is not necessary for learning strong, robust vision encoders. This is a novel insight of the paper.", "weaknesses": "1. For classification tasks, the authors only present ImageNet-1K classification accuracy and do not perform experiments on other popular few-shot and zero-shot classification benchmarks. Including these benchmarks would have strengthened the paper.\n2. Vision-language models like CLIP are also used in text-to-image generation systems. The authors do not address this aspect or present any experiments showcasing the effects of using their encoder in such systems.\n3. This work shares similarities with research that cleans pretraining captions before performing CLIP-style pretraining [1]. However, the authors do not compare their method against such approaches. I believe this comparison is important, as synthetic captions are increasingly used for CLIP-style pretraining.\n4. There is a small reference error in line 235.\n\n\n\n\n[1] Fan, Lijie, et al. \"Improving clip training with language rewrites.\" Advances in Neural Information Processing Systems 36 (2024).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> **W1**. Other popular few-shot and zero-shot classification benchmarks.\n\n**A1**. Thanks for the advice. We have added more evaluation benchmarks, including 10-shot classification on ImageNet-1k, pets and cars, and zero-shot classification on 8 more datasets. \n\n**10-shot classification**  \nWe follow the setting of Cappa[1]. For each dataset and model, we run 3 times, and report the mean results and variance in the following table. Our method surpasses CLIP on IN-1K and Pets by clear margins with improvements of 1.6 and 2.2 points, while being comparable with CLIP on Cars (92.6 v.s 92.7).\n\n| Case        | IN-1K         | Pets          | Cars          |\n|-------------|---------------|---------------|---------------|\n| MAE         | 44.0(0.1)     | 57.7(0.2)     | 32.5(0.1)     |\n| Dinov2      | _77.0(0.1)_   | _94.2(0.1)_   | 76.8(0.2)     |\n| Cappa*      | 70.6(0.2)     | 92.6(0.5)     | 92.2(0.2)     |\n| CLIP        | 75.6(0.1)     | 92.2(0.6)     | **92.7(0.3)** |\n| Superclass  | **77.2(0.1)** | **94.6(0.1)** | _92.6(0.1)_   |\n\n**zero-shot classification**  \nFollowing LiT[2], we train a text encoder on Datacomp-1B by keeping the image encoder locked. In this way, we are able to perform zero-shot classification. We tested the model on the following 10 datasets. Superclass beats openCLIP on seven of the datasets. It is worth noting that SuperClass uses a ViT large model with patch size 16, while the baseline method adopts a patch size of 14. This makes the superiority of SuperClass over openCLIP even clearer.\n\n| Case       | Model    | Data        | Seen samples | IN-1K | imagenet-v2 | imagenet-r | imagenet-a | imagenet-sketch | GTSRB | Rendered SST2 | ObjectNet | SUN397 | Country211 |\n|------------|----------|-------------|--------------|-------|-------------|------------|------------|-----------------|-------|---------------|-----------|--------|------------|\n| openCLIP   | ViT-L/14 | Datacomp-1B | 12.8B        | 79.2  | 72.1        | 90.8       | 69.6       | 68.0            | 58.5  | 61.0          | 74.3      | 74.3   | 31.6       |\n| SuperClass | ViT-L/16 | Datacomp-1B | 12.8B        | 79.7  | 72.4        | 91.6       | 68.8       | 70.6            | 58.5  | 61.6          | 73.9      | 73.8   | 32.3       |\n\n[1] Tschannen, Michael, et al. \"Image captioners are scalable vision learners too.\" NeurIPS 2024.\n[2] Zhai, Xiaohua, et al. \"Lit: Zero-shot transfer with locked-image text tuning.\" CVPR. 2022.\n\n\n> **W2**. The application in text-to-image generation systems. \n\n**A2**. Thanks for the advice. Text-to-image generation is indeed a particularly important application of vision-language models. However, due to the limited time available for rebuttal, we were unable to attempt and train this task. We will explore this issue further in the future. \n\n> **W3**. The effect of synthetic captions\n\n**A3**. Thanks for the nice advice. We use the code provided by LaCLIP[1] https://github.com/LijieFan/LaCLIP for investigation. We compare SuperClass against CLIP with ViT-B/16 following the setting of LaCLIP. The numbers of CLIP and LaCLIP are directly borrowed from the paper. \nAs shown by the results in the following table, Superclass can also benefit from rewritten captions, and the improvement is even greater than that of CLIP (+1.1 vs. +1.6 in zero-shot and +1.2 vs. 1.9 in Linear probing). The possible reason is that the rewritten captions transform the sentence structure but keep the major objects and subjects intact, indirectly enhancing the weight of those visual-related words. \n\n| Case       | Data       | epoch |  Zero-shot | Linear Probing |\n|------------|------------|-------|------------|----------------|\n| CLIP       | CC3M       | 25    | 15.8       | 54.5           |\n| SuperClass | CC3M       | 25    | 16.9(+1.1) | 55.7(+1.2)     |\n| LaCLIP     | CC3M recap | 25    | 21.5       | 56.5           |\n| SuperClass | CC3M recap | 25    | 23.1(+1.6) | 58.4(+1.9)     |\n\n[1] Fan, Lijie, et al. \"Improving clip training with language rewrites.\" NeurIPS 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Thanks to the authors for submitting this work. I found the proposed method simple yet elegant, requiring no data preprocessing, and can be applied at large data scales efficiently. The authors conducted comprehensive experimental results, including added results during rebuttal on few-shot/zero-shot tasks and comparison with related self-supervised/ weakly-supervised methods beyond CLIP.

As noted by the reviewers, there are some minor issues in the current papers: (1) similar features to recent work e.g. CatCLIP (however, I noted this is a concurrent work) and (2) the current contribution of vision backbone pretraining (rather than just CLIP-related) is not emphasized enough in the paper. 

In conclusion, I recommend accepting this paper and hope the authors will continue to address the above-mentioned points in the revised paper.
Decision: Accept (poster)