Classification Done Right for Vision-Language Pre-Training

Authors: zilong  huang
qinghao  ye
bingyi  kang
jiashi  feng
haoqi  fan
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-24 11:30:56

Abstract: We introduce SuperClass, a super simple classification method for vision-language pre-training on image-text data. Unlike its contrastive counterpart CLIP who contrast with a text encoder, SuperClass directly utilizes tokenized raw text as supervised classification labels, without the need for additional text filtering or selection. Due to the absence of the text encoding as contrastive target, SuperClass does not require a text encoder and does not need to maintain a large batch size as CLIP does. SuperClass demonstrated superior performance on various downstream tasks, including classic computer vision benchmarks and vision language downstream tasks. We further explored the scaling behavior of SuperClass on model size, training length, or data size, and reported encouraging results and comparisons to CLIP. https://github.com/x-cls/superclass

Main paper: 
 introduction: 
 Pretraining methodologies [35, 57, 51, 60] that directly harness web-scale image-text dataset have transformed the field of computer vision in recent years. Among them, contrastive language image pretraining (CLIP) [57] has gained escalading popularity and become predominant due to the following reasons. First, it serves as the industry-standard pre-trained model that facilitates zero-shot visual recognition [50, 52] and finetuning on downstream tasks [19, 17] . Second, proper scaling behaviors [12] are observed such that CLIP can consistently benefit from larger models and bigger data to some extent. Third, it offers strong cross-modal abilities as it is inherently designed to understand and connect information across text and images. Therefore, CLIP-style models are the default choices for most modern Visual Language Models [47, 2, 1] , which connect a vision backbone with a deep language model [69, 13] .Despite its success, CLIP necessitates very large batch sizes for training-typically over 64,000-to achieve optimal performance, along with substantial computational resources for text encoding. This high computational demand limits accessibility for researchers with limited resources and engineering expertise. In our work, we aim to address the heavy computational burden by replacing contrastive methodology with a simpler classification approach, eliminates the need for large contrastive batch sizes, and text encoders.In this work, we revisit the classification method for pretraining on large-scale text-image pairs. Some previous works [54, 31, 39, 27, 51] attempt to tackle this by employing bag-of-words classification in a weak supervised learning manner. However, most of these studies have been conducted on a small scale, and there is no evidence demonstrating their scalability in terms of data and model size. In contrast, our method demonstrates the performance of SuperClass on a scale comparable to CLIP [57] , achieving favorable model performance with 13 billion seen samples on 1 billion unique text-image pairs. Some other concurrent efforts [31] have also attempted to replace contrastive learning with classification. However, they rely heavily on preprocessing the text modality, using bag-of-words and other hand-crafted rules to convert text into semi-labels. Some common practices include filtering, word segmentation, lemmatization, and the removal of numbers and stopwords to create a unique vocabulary of clean words. We found the preprocessing often eliminates long-tailed words or stopwords that contain valuable information for representation learning (see Sec. 4.4). In contrast, SuperClass simply utilizes raw word tokens as supervision signals without requiring any hand-crafted preprocessing: no filtering or removal of stopwords. Hence SuperClass preserves all information from the original text descriptions as supervision signal.We proposed a Super-simple-Classification approach (SuperClass) that simply trains to classify raw text tokens and scales as good as CLIP. As shown in Figure 1 , similar to CLIP, SuperClass directly operate on text tokens with any manual text filtering. Our comprehensive empirical study shows that even without the need for a text encoder, classification methods can achieve performance comparable to the contrastive approach in terms of both model capabilities and data scalability. We demonstrate that SuperClass is a competitive alternative to its contrastive counterpart on both image classification and vision & language tasks. Pretrained on the same Datacomp-1B [21] datasets with an equal number of seen samples, SuperClass dominantly outperforms its contrastive counterparts across various of vision only and vision & language scenarios. We further explore the scaling behavior of SuperClass concerning model size and number of seen samples. Experiments suggest that classification-based methods can exhibit competitive or even superior scaling behavior compared to their contrastive counterparts. We hope our method, experiments and analysis can encourage future potentials of classification-based methods as the foundational vision-language pretraining methods.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes] Justification: refer to abstract and introduction Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
related work: 
 With the growing availability of large-scale, web-sourced image-text datasets [57, 65, 6, 68, 21, 63, 4, 62] , new methods have emerged to leverage this data as supervision for training deep representations. These approaches typically involve one of three strategies: using text as classification labels, implementing image-text contrastive learning, or treating text as autoregressive targets.Text as classification labels. The exploration of image-text data for model training has deep roots, with early work like Image-to-Word [54] over two decades ago aiming to enhance content-based image retrieval. This study pioneered efforts to train models to predict nouns and adjectives in text documents linked to images. Building on these early ideas, subsequent research has sought to improve data efficiency [68, 43] , model effectiveness [31, 27] , and vocabulary expansion [27, 78, 51, 39] . With the recent develop of network architecture, Tag2Text [27] and RAM [78] have employed Vision Transformers (ViT) [18] as vision backbones, extracting nouns from the CC12M dataset [6] and, through a combination of rules and manual selecting, arrived at 6,449 words to use as classification categories. Similarly, CatLIP [51] has filtered out "gold labels" from the CC3M [65] and Datacomp-1B [21] datasets based on certain rules, and then trained visual models using even larger-scale image-text pair datasets.Unlike previous classification methods that rely on complex rules or manual filtering to curate "gold labels" for classification vocabularies, our approach eliminates the need for such filtering. Instead, we directly leverage text tokens as classification categories, preserving valuable textual information that might otherwise be discarded.Image-text contrastive learning. Large-scale contrastive vision-language pretraining gained traction with the introduction of CLIP [57] and ALIGN [30] . Since then, numerous approaches have focused on enhancing CLIP's performance [76, 45, 44, 11, 7, 74] . For instance, SigLIP [76] reduces the computational load of CLIP's softmax-based contrastive loss by employing a sigmoid loss for local pairwise similarity calculations. LiT [77] adopts pretrained vision and language backbones with contrastive training, while other methods [45, 44, 19] aim to enhance training efficiency in image-text pretraining. InternVL [11] further innovates by integrating a large language model as the text encoder within CLIP.In our approach, we challenge the necessity of an additional backbone to encode text information for contrastive learning. Instead, we directly use text token input as the supervisory signal, eliminating the need for text encoding and avoiding the computational overhead of large contrastive operations. This streamlined setup achieves performance comparable to dual-backbone methods.Text as autoregressive targets. Various recent studies [15, 61, 38, 32, 41, 26] have delved into employing image captioning for model pretraining. SimVLM [71] has innovated this field by pioneering the pretraining of a multimodal encoder-decoder that fuses vision and language at an early stage, leveraging a hybrid architecture for applications such as visual question answering (VQA). CapPa [70] demonstrates that a simple encoder-decoder setup can efficiently pretrain vision encoders solely through captioning. Furthermore, recently studies [75, 42, 37] combine contrastive learning with captioning objectives, occasionally incorporating an additional text encoder.In this work, we revisit the classification-based approach using large-scale visual-language datasets. Unlike the image captioning methods mentioned earlier, our classification method integrates the text captioning decoder within the vision encoder, allowing a single vision encoder to connect both modalities. Experiments demonstrate that SuperClass achieves competitive, and often superior, performance across various downstream tasks.Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Discussed in the conclusion Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
a simple classication approach to pretrain vision encoders: 
 In this section, we present our proposed approach, SuperClass, which employs a classificationbased pretraining method using text supervision. We begin by outlining the general framework of SuperClass. Next, we explain how text is converted into category labels without the need to select "gold labels", allowing all text to supervise the training of the image encoder. Finally, we illustrate our choice of loss design among various classification losses. Additionally, recognizing the differing significance and discriminative power of each word, we incorporated inverse document frequency as class weights in the loss design.Overview. We aim to establish a pretraining method based on image classification that matches CLIP in simplicity, scalability, and efficiency. To achieve this, we follow the standard protocol by utilizing Vision Transformer (ViT) backbones as vision encoders, followed by a global average pooling layer and a linear layer as the classification head to output the logit vector x. The supervision targets are derived from the text associated with the image, and the classification loss is computed using the text-derived classification labels and the predicted logits.Texts as Labels. We directly use tokenized text as K-hot labels, where K is the number of tokens in the given sentences. More specifically, for a given image-text datasetD = {(I i , T i ) | i ∈ [1, N ]}with N pairs of images I and text captions T , we differ from previous classification-based methods by directly using an existing subword-level tokenizer, such as the one used in CLIP or BERT, with a vocabulary size V . This tokenizer inputs the text T and obtain the set C of corresponding subword IDs, which serves as the classification labels. The label in the set C satisfies {c ∈ [1, V ]}. The classification labels C will be converted into K-hot vector y ∈ R V , where y c = 1 when c in the set C, otherwise y c = 0.Compared to previous methods, our approach does not require any preprocessing or manual threshold setting, making it straightforward. At the same time, it also avoids the out-of-vocabulary issue that might be encountered by previous approaches.Classification Loss. A significant body of research has focused on multi-label classification loss. However, it is important to emphasize that our primary goal is to pretrain vision encoders rather than prioritize multi-label classification accuracy. In a multi-label scenario, a Softmax loss is applied in SuperClass by representing labels in a probabilistic manner, where ŷc is a normalized weighted label.EQUATIONWe evaluated several multi-label classification losses, including Softmax loss, BCE loss, soft margin loss, ASL [59] , and two-way loss [33] . Surprisingly, the simple Softmax loss yielded the best pretraining results. This may be due to the fact that existing multi-label classification losses assume that labels are precise and exhaustive, aiming to optimize the margin between positive and negative classes. However, the inherent noise in image-text data and the limitations of text in fully capturing an image's content mean that not all objects in an image are always referenced in the accompanying text.Inverse Document Frequency. Within the subword vocabulary, not all categories contribute semantically equally, as different words carry varying amounts of information. Additionally, the subword dictionary contains many words unrelated to visual content that frequently appear in sentences, which do not provide effective supervisory information. Therefore, words with higher information content should be given greater weight during training. To achieve this, we employ Inverse Document Frequency (IDF) as a measure of information significance. The fewer the number of samples containing a specific word, the stronger its ability to differentiate between samples. We use the IDF statistic of each category (subword) as the weight for the corresponding classification label, assigning different weights w c to the classification labels c.EQUATIONwhere |D| denotes the total number of image-text pairs, df(c) is the document frequency (df) of subword c, in other words, it's the count of texts containing subword c. For greater ease of use, we have implemented an online IDF statistic that is computed during the training process, eliminating the need for pre-training offline statistics. This approach enhances the user-friendliness and portability of our method.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Justification: Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced. • All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition. • Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material. • Theorems and Lemmas that the proof relies upon should be properly referenced. 
experiment setup: 
 We use a standard subset of the datacomp dataset [21] for pre-training, which contains about 1.3B image-text pairs. A batch size of 16k and 90k are adopted for our classification models and CLIP models. In the ablation section, all experiments are conducted with a batch size of 16k. To make a fair comparsion with the CLIP, we use 90k batch size, adopt the AdamW with a cosine schedule, and set the same learning rate and decay as CLIP.In order to better highlight the effectiveness of pretraining method, we concentrate on the properties of the frozen representations.Linear probing We evaluate the classification accuracy when using the full ImageNet-1k [14] training set to learn a dense projection layer and frozen the parameters of backbone. We follow the linear probing training recipe from MAE [24] . 10-shot classification Following the setting of Cappa [70] , we perform 10-shot classification on ImageNet-1k [14] , Pets [56] and Cars [34] . For each dataset and model, we run 3 times, and report the mean results and variance.Locked-image Tuning Locked-image Tuning (LiT) [77] employs contrastive training to align locked image and unlocked text models. Generally, LiT is an efficient way to equip any pretrained vision backbone with zero-shot classification and retrieval capabilities. We follow the setup from LiT [77] and assess the zero-shot classification accuracy on ImageNet-1k [14] .Collaborating with language models Motivated by recent works [47, 11, 1, 67, 2, 71] combining pretrained vision backbones [57, 19, 76] and language models [69, 13] , we investigate the amenability of the learned representations to interface with a text decoder. Here, we evaluate two ways to collaborate with large language models. 1) following ClipCap [53] , we frozen both pretrained image encoder and pretrained 12-layer GPT2 decoder [58] , only train an adapter to connect the image encoder and language model to perform image captioning on COCO captions [9] . 2) following LLaVA [47] setup, we train and finetune a projection layer and a pretrained large language models, Vicuna-7B [13] to solve downstream tasks, including VQAv2(val) [22] , GQA [28] , VizWiz(val) [23] , T-VQA(val) [66] , SQA(img) [79] , MMBench(en) [48] , MME [20] , POPE [46] , MMMU [25] and SEEDBench [40] .Comparison with different types of pretraining methods In Table 1 , we compare the models trained by SuperClass with the different types of pretraining methods, including contrastive or clustering based methods [10, 5, 80, 55] , reconstruction based [3, 73, 8, 24] , and vision-language pretraining based methods [57, 70, 21] . In general, the proposed method achieves best performance among these pretraining methods.Compared to the current SOTA self-supervised model DINOv2 [55] , our method achieves a 0.5% higher accuracy in IN-1K linear probing (85.0 vs 84.5) without a bunch of bells and whistles.Although SuperClass has seen more samples, it operates as a simpler classification framework and does not employ MultiCrop, Masked Image Modeling, or Contrastive learning, as DINOv2 does.Although comparing a self-supervised learning method to a (weakly) supervised learning approach is a system-level comparison, we still observe that a simple classification pretraining approach demonstrates superior performance across many classy benchmarks that shown in Table 2 .Compared to the contrastive counterparts CLIP [57] , our method achieves higher linear probing top-1 accuracy on ImageNet-1K dataset with ViT-Base (80.2 vs 78.5) and ViT-Large (85.0 vs 82.7) as backbone. For a fair comparison, we further make a comparison with OpenCLIP [29] which trains a ViT-Large model with a batch size 90k based on Datacomp-1B dataset. Our method consistently outperforms OpenCLIP by a large margin (85.0 vs 83.9). In Further comparison with CLIP In Table 3 , we compare the models trained by SuperClass with the currently widely used CLIP models, including zero-shot classification, and COCO captioning.To verify the effectiveness of the pretraining method, we adapt the standard ViT [18] structure as the visual backbone and added a classification head on top of it. We use the open-source Datacomp-1B [21] dataset and encounter 13B samples in the training process. With Lock image Tuning [77] , the trained classification model also gains the ability of zero-shot classification. Our method achieves 79.7% Top-1 zero-shot accuracy on ImageNet-1k datatset which is much better than OpenAI CLIP ViT-L/14 and OpenCLIP ViT-L/14. Although maybe they are not directly comparable, this do reflect that the vision model trained by the proposed SuperClass is with strong visual perception capabilities.Combining the frozen vision encoder with a frozen pretrained 12-layer GPT-2 decoder [58] via a trained adapter, the models are trained on COCO captions [9] and CIDEr socres are reported. We observe that the CIDEr socres of our method are slightly below OpenAI's CLIP, which may be due to the use of different datasets; OpenAI's CLIP utilizes an internal dataset, WiT-400M. However, our approach outperforms our implemented CLIP model with the same settings.Overall, the models trained by proposed SuperClass demonstrated marginally improved accuracy in both classification capabilities and the vision & language task when compared to the contrastive pretrained CLIP models.Large multi-modal models Many large multi-modal models integrate pre-trained vision backbones with large language models. We explore how amenable the learned representations are to interfacing with a text decoder. Following the LLaVA setup [47] , we combine frozen CLIP models and SuperClass models with the pretrained Vicuna-7B [13] and perform downstream tasks. In Figure 2 , we show some results of vision&language downstream tasks. The results demonstrate that SuperClass models could achieve better performance than CLIP models on the majority of datasets. It is worth mentioning that, in comparison to CLIP models, SuperClass models exhibit significantly better performance on VQAv2 [22] , T-VQA [66] , and MMBench [48] , which pertain to OCR and fine-grained recognition tasks, respectively. In addition, the overall accuracy measurement on VizWiz [23] are not stable due to a significant portion of questions being labeled as unanswerable. To ensure the completeness of our findings, we still present the results on this dataset.Due to space limitations, detailed numerical results are provided in the appendix. Additionally, you can find more experimental results in the appendix. In the top row of Figure 2 , we showcase the performance across classification and vision & language tasks for varying model scales. For a fair comparison, both CLIP and SuperClass models undergo training with identical settings, which include a batch size of 16k and 512 million seen samples. As shown in Figure 2 , with the model scaling up, we observe a corresponding enhancement in performance, whether it is for classification tasks or the downstream tasks associated with LLaVA. Generally speaking, with the same model size, models pre-trained using SuperClass exhibit superior precision compared to those trained with CLIP. SuperClass demonstrates better scaling on zero-shot classification and VQAv2, T-VQA tasks.In the bottom row of Figure 2 , we showcase the performance across classification and vision & language tasks for varying seen samples. For a fair comparison, both CLIP and SuperClass models undergo training with identical settings, which include a batch size of 16k and ViT-L/16 as backbone. Figure 2 illustrates that as the number of seen samples grows, there is a noticeable improvement in performance for both classification and downstream tasks linked to LLaVA. Typically, models pre-trained with SuperClass outperform those trained with CLIP in terms of accuracy when given the same amount of seen samples. SuperClass exhibits the same or slightly better scaling behavior compared to CLIP on downstream tasks. In addition, SuperClass does not require a text encoder, it offers better efficiency in training compared to CLIP.To verify the rationality of the SuperClass, we conduct extensive ablation experiments that pretrain on datacomp-1B [21] and evaluate on several downstream tasks with different settings for SuperClass.Word-level tokenizer vs. Subword-level tokenizer Table 4 presents the results of word-level tokenizer and subword-level tokenizer on serval daownstream tasks. We use the tokeinzer in openai CLIP as our subword-level tokenizer. We compare it with the word-level tokenizer used in CatLIP [51] , which carefully selected approximately 40,000 "gold labels" from the datacomp-1B dataset. Aside from the tokenizer being different, all models are trained under the same settings.For ViT-S/16, word-level tokenizer achieves better classification accuracy than subword-level tokenizer. A possible reason is that when the model capacity is limited, the filtered clean supervisory information may be more conducive to model convergence. However, with the increasing size of the model, subword-level tokenizer gradually outperforms the word-level tokenizer, whether in classification tasks or vision & language tasks.Regardless of model size, on most vision & language tasks, subword-level tokenizer tends to perform better than word-level tokenizer. The reason may be that subword-level tokenizer retain a substantial amount of language-related information, although it may not be highly relevant to visual information. This makes the features of the models trained with the subword-level tokenizer more readily integrated with large language models.Overall, using a subword-level tokenizer exhibits better scaling behavior and is more suitable for use in large multi-modal models.Different subword-level tokenizers Table 5 presents the results on classification tasks and LLaVA downstream tasks with different subword-level tokenizers. Here, we compare the character-based byte pair encoding tokenizer [64] used in CLIP [57] , the WordPiece [72] tokenizer used in BERT [16] and the SentencePiece [36] tokenizer used in LLama [69] , they are all subword-level tokenizers. The tokenizer used in openai CLIP obtains best performance on the classification task and LLaVA downstream tasks. Finally, we choose the tokenizer used in CLIP [57] for the training of SuperClass models.Classification loss Table 6 represents different classification loss on ImageNet-1k dataset. We selected several of the most commonly used multi-label classification losses for experimentation. Softmax loss is often used in single-label classification tasks. It is possible apply a softmax loss in a multi-label scenario through describing labels in a probabilistic way. BCE loss is a binary cross-entropy (BCE) loss and is often used in multi-label classification tasks as baseline. Asymmetric Loss(ASL loss) a improved BCE loss to address positive-negative imbalance. Soft margin loss is a margin-based loss for multi-label classification tasks. Two-way loss is the current state-of-the-art (SOTA) for multi-label classification tasks. Surprisingly, the simplest softmax loss outperforms all other multi-label classification losses by a large margin. We believe that existing multi-label classification losses operate under the assumption that labels are both accurate and complete, aiming to optimize the classification margin between positive and negative classes. However, in reality, image-text data contains considerable noise, and a single text passage cannot possibly capture all the contents of an image. Consequently, certain categorical objects present in the image may not be mentioned in the associated text. In the context of image-text pretraining, how to design a better loss function remains a question worthy of exploration.Considering that the importance of each category (subword) in the vocabulary is not equal and the information they carry varies, we use IDF as class weights. The Table 7 repsents the results of with and without IDF as class weights. SuperClass without IDF experienced a noticeable decrease in accuracy on classification tasks, the change in precision on LLaVA tasks is not significant.Removing stopwords? Stop words are commonly used in Text Mining and Natural Language Processing (NLP) to eliminate words that are so widely used that they carry very little useful information. In the previous classification methods, the stopwords are removed. The stopwords are download from NLTK [49] . However, the results in Table 7 shows that the keeping stopwords could help the vision encoder to gain better performance on classification tasks.Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes] Justification: Important details are provided in the experiment section. 
limitation and conclusion: 
 We have conducted a thorough comparison of vision encoders pre-trained with contrastive and classification objectives and determined that models pre-trained with classification surpass CLIP models in both classification and vision language tasks. Additionally, our approach does not require a text encoder, which leads to higher training efficiency than that of CLIP. Furthermore, our findings suggest that classification as a pre-training task may have beneficial scaling properties as model and data sizes increase, and we encourage future research to delve into this possibility.While it delivers impressive results on various downstream tasks, it completely ignore word order and object relationships, which implies that we are losing important supervisory information. Addressing this will be the direction of our future research efforts.To sum up, we have demonstrated that straightforward image classification can serve as an effective pre-training strategy for vision backbones derived from image-text data. Our aim is to stimulate subsequent studies to pay more attention to the benefits of classification as a pre-training task for vision encoders.Table 10 : Performance of frozen visual representations trained via image classification (SuperClass) and constrastively (CLIP). Linear probing and zero-shot classification are both tested on ImageNet-1k dataset. Captioning is conducted on COCO captions and CIDEr is reported in the table. The zero-shot accuracy of SuperClass is obtained after lock-image tuning [77] .Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We use public data for model training. The code will be cleaned and made public in the future.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: we provide all the details in the experiment section.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [No] Justification: We follow the convention in prior works and report the performance number on the standard benchmarks.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: please refer to the experiment section.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: We discuss in the conclusion.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA]Justification:Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We used public datasets.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: • The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not. • If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable. • Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed. • While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. , with an open-source dataset or instructions for how to construct the dataset). (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Justification:Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Laion's CLIP https://huggingface.co/laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K. 


Decision: Thanks to the authors for submitting this work. I found the proposed method simple yet elegant, requiring no data preprocessing, and can be applied at large data scales efficiently. The authors conducted comprehensive experimental results, including added results during rebuttal on few-shot/zero-shot tasks and comparison with related self-supervised/ weakly-supervised methods beyond CLIP.

As noted by the reviewers, there are some minor issues in the current papers: (1) similar features to recent work e.g. CatCLIP (however, I noted this is a concurrent work) and (2) the current contribution of vision backbone pretraining (rather than just CLIP-related) is not emphasized enough in the paper. 

In conclusion, I recommend accepting this paper and hope the authors will continue to address the above-mentioned points in the revised paper.

Reviews: review: paper_summary: paper_summary: This paper introduces a multi-label classification pre-training style for visual image encoder pre-training.
strength_weakness: strength_weakness: - The zero-shot capacity of such a multi-label pre-trained model is not well demonstrated.
- The paper lacks a comprehensive comparision with weakly-supervised or unsupervised visual encoder methods. Only comparing with CLIP is not enough.
- Some arguments about previous so-called "bag-of-word classification" pre-trained methods may not be correct.
questions: questions: - 1. The greatest advantage of CLIP is its zero-shot capacity across various downstream tasks. The zero-shot ability of SuperClass is not well-demonstrated on downstream tasks like zero-shot text-image retrieval, zero-shot text-video retrieval, and zero-shot STR. The reviewer thinks the authors should include an analysis of the zero-shot ability like the original CLIP paper.

- 2. The baselines should not be CLIP-style pre-trained vision language models. The proposed method aims to pre-train a vision transformer. The baseline should be other weakly-supervised methods or self-supervised methods like MAE, etc. The authors should compare these methods in terms of training efficiency and performance of transfer learning. These methods also show good transfer learning ability, for example, MAE has good performance of transfer learning on COCO, while it is only pre-trained on ImageNet-1K.

- 3. Some arguments about previous so-called "bag-of-word classification" pre-trained methods may not be correct. In the introduction, the authors claim " However, these methods fail to gain popularity from the community, as most of the experiments are conducted on a small scale and there is no evidence showing their scalability to data size and model size in comparison to CLIP ". Nevertheless, CatCLIP[1] already conducts experiments with CLIP-H on DataComp-1.3B.
    - 3.1. The authors should also compare with these "bag-of-word classification" pre-trained methods.

- 4. The authors say "All experiments were carried out on an A100 GPU equipped with 80GB of memory." Could the authors provide training times for the experiments in Table 1? Besides, are experiments in Table 2 also trained on an A100 80G? If so, could the authors provide the training time? 

[1] https://arxiv.org/pdf/2404.15653
limitations: limitations: Yes.
paper_summary: paper_summary: This paper explores a new direction to pretrain vision backbones using large scale image-text pairs for learning visual representations which are suitable to various downstream tasks. More specifically, this work proposes a classification based objective function as an effective alternative to CLIP's standard cross-modality similarity based constrastive loss. The proposed model SuperClass uses a image encoder to map image to token probabilities having a head size equal to CLIP's tokenizer vocabulary. The texts are converted into labels using weighted tokenizer. Both CLIP baseline and SuperClass are pretrained on datacomp image-text pairs dataset and evaluation results across various vision and vision-language benchmarks are reported. The proposed method shows greater efficiency due to being text-encoder free and also performs favorably well over previous approaches. Extensive ablation studies are performed to justify the design choices made in the paper.
strength_weakness: strength_weakness: **Weaknesses**
1) In my understanding, one of the weaknesses of this work is lack of comparisons with related works. For example, the only method with which SuperClass is compared with is CLIP which is only a baseline. I believe there should be comparisons with other related SOTA works. 
2) The proposed approach might not be capable of doing multi-label classification or zero-shot classification as compared to other competitors such as RAM, CLIP etc. Also CLIP has out of the box additional features such as prompt ensembling, zero-shot segmentation [1] etc which might be not present in SuperClass. This would question how it can comprehensively show effectiveness over CLIP and other vision-language models.
3) It is not clear how the proposed method learns suitable representations when using the subwords as labels which does not correspond to any visual concept. I believe there is a bit of analysis missing in the manuscript.
4) The results for the zeroshot and linear probing in Tab1 and Tab 2 of the main paper are different for the same model. It is unclear if why the results are different for same model.


[1] Extract Free Dense Labels from CLIP (ECCV 2022)
questions: questions: Please refer to the weaknesses section for the questions. I will highly recommend the authors submit a rebuttal response. I will be happy to reconsider my final scores based on the response from the authors.
limitations: limitations: Yes the authors have adequately addressed the limitations in the paper.
paper_summary: paper_summary: This paper proposes a simple classification-based vision-language pretraining method. The proposed SuperClass approach directly uses an off-the-shelf subword-level tokenizer to obtain the classification labels from raw text, without requiring any preprocessing. Then, the vision encoder is trained by optimizing the multi-label softmax loss, with Inverse Document Frequency (IDF) as the weight of each label. SuperClass achieves promising performance on classification and various vision-language tasks. Ablation experiments are conducted to validate the impact of different design choices.
strength_weakness: strength_weakness: 1. The robustness of the proposed method to different model types remains unclear. All the experiments in this paper use ViT as the vision encoder, and there is no evidence to demonstrate the effectiveness of the SuperClass on other encoder architectures, such as ResNet.
2. The ablation experiment on different classification losses is only conducted on classification tasks, using a ViT-B/16 backbone and 512M seen data samples. It is important to demonstrate the robustness of the softmax loss on a broader range of vision-language tasks, beyond just classification. Furthermore, the impact of the choice of loss function on the scalability of the proposed method is not discussed.
questions: questions: Please refer to the weaknesses.
limitations: limitations: The authors discuss some of the limitations of their work in Section 5. But I would like them to consider some of my concerns above.
paper_summary: paper_summary: The paper proposes a simple alternative to CLIP-style pretraining that doesn't require a text encoder and can be done using only a text tokenizer. The goal is to provide a simpler yet more efficient alternative to vision-language model (VLM) pretraining, which is known to be very expensive. Additionally, the authors simplify the VLM pertaining setup into a classification task which is both novel and very intuitive. 
The authors demonstrate the efficiency of their method through experiments on both classification and vision-and-language downstream tasks. Additionally, they present a comprehensive set of ablations to dissect the performance gains of their proposed approach.

While there are some limitations to their proposed method, the simplification offers valuable benefits for traditional classification and vision-language tasks.
strength_weakness: strength_weakness: 1. For classification tasks, the authors only present ImageNet-1K classification accuracy and do not perform experiments on other popular few-shot and zero-shot classification benchmarks. Including these benchmarks would have strengthened the paper.
2. Vision-language models like CLIP are also used in text-to-image generation systems. The authors do not address this aspect or present any experiments showcasing the effects of using their encoder in such systems.
3. This work shares similarities with research that cleans pretraining captions before performing CLIP-style pretraining [1]. However, the authors do not compare their method against such approaches. I believe this comparison is important, as synthetic captions are increasingly used for CLIP-style pretraining.
4. There is a small reference error in line 235.




[1] Fan, Lijie, et al. "Improving clip training with language rewrites." Advances in Neural Information Processing Systems 36 (2024).
questions: questions: 1. In Table 7, the authors show that removing stop-words has no effect and state in line 279 that "keeping stopwords could help the vision encoder." This statement is not explained and seems counter-intuitive. Could the authors elaborate on this observation?
2. The authors claim they can use IDF weights in an online manner. Could the authors explain how they achieve this? I don't understand how this process can be done online.
3. The authors demonstrate that the performance of SuperClass is as good as or better than traditional CLIP-style pretraining on ImageNet-1K classification. However, it would be interesting if the authors could specify which classes benefit more from their method compared to traditional CLIP. Such an analysis could help understand where the gains stem from.
limitations: limitations: 1. The authors acknowledge that their approach completely ignores word order, which can significantly impact the encoder's ability to understand tasks requiring spatial reasoning.
2. Additionally, this approach may not be suitable for text-to-image systems, as these models need to understand word order and infer relationships from text to generate accurate images.

Comments: title: Discussion
Rebuttal by Authors Part 4
Discuss
Rebuttal by Authors Part 3
Discussion
Thank you for providing the rebuttal response
Rebuttal by Authors Part 2
Rebuttal by Authors Part 2
comment: Thank the Reviewer aefc for the feedback.

We have addressed all Reviewer aefc's questions in our previous responses, including but not limited to
- performance and scalability

    > Line #212 subsection "Data Scaling results Line" and #221 subsection "Model scaling results" in the paper

    > The results of Zero-shot classification, VLM downstream tasks ... in https://openreview.net/forum?id=Hd2EOwKItm&noteId=JxjyAuZo5M (Rebuttal by Authors)

    > Few-shot classification in A1 https://openreview.net/forum?id=Hd2EOwKItm&noteId=1hHvBn6szv (Response to Reviewer bBvT)

- comparisons and differences with CatLIP[46]

    > A3 in https://openreview.net/forum?id=Hd2EOwKItm&noteId=JxjyAuZo5M (Rebuttal by Authors) "CatLIP [46] is a concurrent work. We have already cited and compared it in the paper. "

    > Line #69 and Line #233 subsection "Word-level tokenizer vs. Subword-level tokenizer" in the paper 

    > Fine-tuning results in A4 https://openreview.net/forum?id=Hd2EOwKItm&noteId=OOhKhswjjl (Rebuttal by Authors Part 2)

    > https://openreview.net/forum?id=Hd2EOwKItm&noteId=KeV1R5CtDE (Rebuttal by Authors Part 3) "The differences from other...classification-based pretraining methods are also evident...manually curated "golden labels"..."

- Performance comparisons with CLIP and Dinov2
   
    > The results of Zero-shot classification, VLM downstream tasks ... in https://openreview.net/forum?id=Hd2EOwKItm&noteId=JxjyAuZo5M (Rebuttal by Authors)

    > The results of 10-shot classification in A1 of https://openreview.net/forum?id=Hd2EOwKItm&noteId=aVM6gFdSHR (Response to Reviewer bBvT)

    ...

- Different data from CLIP paper

    > https://openreview.net/forum?id=Hd2EOwKItm&noteId=ogRsx4SgwE (Rebuttal by Authors Part 4) "Regarding Q1, we utilized the open-sourced checkpoint [1] from paper [2]...This serves as a stronger baseline compared to the original OpenAI CLIP."

We do not intend to engage in repetitive responses and hope to strengthen and solidify our work based on the reviewers' valuable feedback.

We further thank all the reviewers.
Thank the authors for the response.

## 1. "performance and scalability"
***
Well, the authors say that your methods have better "performance and scalability" in https://openreview.net/forum?id=Hd2EOwKItm&noteId=KeV1R5CtDE **A2**.
So,
- The reviewer provides examples that previous methods have better "performance and scalability".
- The proposed method use **10x** data, but is close to DINOv2 with the same parameters.
-  The reviewer does not even mention similar work CatCLIP[1], which also trains CLIP in a classification way. It also demonstrates better "performance and scalability".

## 2. It seems the reviewer suggests that foundational modeling research might be redundant when the absolute performance are weaker
***
The authors are trying to mislead readers.

- The paper does not propose something novel, CatCLIP[1] already does the classification pre-training job.
- The performance/scalability of the paper is no better than recent literature [2,3].

The proposed method does not demonstrate its necessity when there are many other strong self-supervised/weakly-supervised/language supervised (may belong to the weakly-supervised class) methods, so it is reasonable that the reviewer thinks this work may not be necassary for the community.

## 3. Different data from CLIP paper

The authors do not answer this question. https://openreview.net/forum?id=Hd2EOwKItm&noteId=vqnfjpW7aT Q1.

```
Could you please tell me where the zero-shot 46.9 COCO Text-to-Image result comes from? I check the paper of CLIP[1], the zero-shot image retrieval result is 37.8.
```



## 4.
Finally, as an independent reviewer, the reviewer thinks he can have his own opinion and rating about the paper.


Best,

Reviewer aefc

[1] https://arxiv.org/pdf/2404.15653

[2] https://arxiv.org/abs/2303.15389

[3] DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024.
We thank reviewer aefc for their timely response.

Regarding Q1, we utilized the open-sourced checkpoint [1] from paper [2], which employs ViT-Large as its backbone and is trained with Datacomp-1B and 12.8 billion seen samples. This serves as a stronger baseline compared to the original OpenAI CLIP.

>**Reviewer aefc commented "The reviewer does not know why we need such a method when we have many other ... pre-trained ... large models" (VLMs/self-supervised/weakly supervised).**

It seems the reviewer suggests that foundational modeling research might be redundant when the absolute performance are weaker compared to a "large model" pretrained in a systematic way. The reviewer appears to prioritize "higher numbers" even when comparing in an apple-to-orange setting. While we acknowledge that a big model with higher numbers is often advantageous from a downstream users' perspective, we believe that a scientific foundational model researcher would in favor of careful ablations rather than chasing higher numbers with biggest models.

For instance, the reviewer argues that SuperClass cannot demonstrate better "performance and scalability" than DINO v2, citing that the DINO v2 *ViT-g 448* model, pretrained on a customized dataset at 448 resolution - superior to our largest *ViT-L 224* model at 224 resolution. 

We appreciate the reviewer's observation that our *ViT-L 224* resolution model is less powerful compared to a *ViT-g 448* resolution model. It appears that the reviewer has overlooked various systematic differences - unintentionally or for some other unknown reason - comparing scientific papers in an apple-to-orange manner.

>**The reviewer aefc further suggested "Overall, the baselines are far from strong. "**

The baselines we compared here are CLIP trained with DataComp [2] and DINOv2 [3], which are important milestone papers accepted at NeurIPS 2023 and TMLR 2024, respectively. For some unknown reason, the reviewer believes CLIP, DataComp and DINOv2 are far from strong. We kindly respect the reviewer's false claim.

We would like to wish the reviewer the best, and politely disagree with the reviewer's comments and rating.

We further thank other reviewers for their professional, scientific, objective reviews.

[1] https://huggingface.co/laion/CLIP-ViT-L-14-DataComp.XL-s13B-b90K

[2] DATACOMP: In search of the next generation of multimodal dataset, NeurIPS 2023.

[3] DINOv2: Learning Robust Visual Features without Supervision, TMLR 2024.
Dear authors:

Thank you for you response.

- Q1
***
Could you please tell me where the zero-shot 46.9 COCO Text-to-Image result comes from? I check the paper of CLIP[1], the zero-shot image retrieval result is 37.8.

- Q2
***
- CLIP is born in  26 Feb 2021. Many other VLM pre-trained under language supervision emerged, e.g., EVA-CLIP[2]
- Self-supervised pre-training and weaky-supervised methods are also hot. According to the provided information, the proposed method uses 10x data (12.8B), and it is close to DINOv2(1.92B). If we talk about "performance and scalability", DINOv2 ViT-g/14@448 achieves 86.7% linear probing results on ImageNet-1K.

Overall, the baselines are far from strong. The reviewer does not know why we need such a method when we have many other powerful and versatile pre-trained VLMs/self-supervised large models/weakly supervised large models.

I still retain my score at this time.

Best,

Reviewer aefc

[1] https://arxiv.org/pdf/2103.00020

[2] https://arxiv.org/abs/2303.15389
Thank you for your insightful feedback and for recognizing the strengths of our paper. We appreciate your thoughtful comments on the potential applications and limitations of our model. Your suggestions provide valuable directions for future work.

Thank you again for your comprehensive review and for increasing the score. Your support and recognition are greatly appreciated.
Thank you for your quick response and for taking the time to review our paper. We appreciate your feedback and are grateful for your recognition of our work.
Thank you for your valuable reviews and feedback.

We appreciate your insights on emphasizing the new pretraining style for vision backbones rather than comparing it predominantly with CLIP. We will ensure that the final version of the paper includes all the rebuttal discussions and places greater emphasis on vision backbone pretraining to eliminate any ambiguities.

Thank you again for your constructive comments and for increasing your score to a weak accept. We hope that our paper will contribute positively to the research community.
Thanks for your reviews and feedback.
>**Q1:** So far, we only see zero-shot performance on classification tasks. It seems that CLIP training style shows broader zero-shot ability in different downstream tasks.

**A1:**  We have evaluated zero-shot retrieval on COCO dataset and zero-shot segmentation on Pascal context and COCO stuff (see more details in the response to Reviewer W9bX). The experimental results show that Superclass could achieve competitive performance compared to CLIP.
| Case | Model | COCO Image-to-Text |  | COCO Text-to-Image |  |
|:---:|:---:|:---:|:---:|:---:|:---:|
|  |  | R@1 | R@5 | R@1 | R@5 |
| openCLIP | ViT-Large | 62.6 | 84.2 | 46.9 | 70.7 |
| SuperClass | ViT-Large | 62.1 | 83.3 | 47.1 | 70.7 |

| Method | Backbone | #Seen sample | PASCAL Context | COCO Stuff |
|---|---|---|---|---|
| CLIP | ViT-B/16 | 1.28B | 16.2 | 8.7 |
| Superclass | ViT-B/16 | 1.28B | 20.2(+4.0) | 13.2(+4.5) |

Besides, we also present results of linear probing and fine-tuning on ImageNet-1K. Compared to CLIP, SuperClass outperforms openCLIP on seven out of ten zero-shot classification datasets. Our method achieves a 1.1% higher accuracy in IN-1K linear probing (85.0 vs 83.9). To further demonstrate the transfer ability of our method, we conducted semantic segmentation on ADE20K and instance segmentation on COCO. Moreover, we provide experimental results on various vision & language tasks after integrating with large language models. The results show that our method outperforms CLIP.
 
We hope that by showcasing our competitive performance in zero-shot retrieval, zero-shot segmentation, linear probing, fine-tuning, semantic segmentation,  instance segmentation, and various vision & language tasks, we can address the reviewer’s concern of "only see zero-shot performance on classification tasks".

>**Q2:** From the comparison, I do not see the superiority of the proposed pre-trained method compared to self-supervised learning, especially since the proposed method is actually weakly-supervised. 

**A2:**  Compared to the current SOTA self-supervised model DINOv2, our method achieves a 0.5% higher accuracy in IN-1K linear probing (85.0 vs 84.5). Although SuperClass has seen more samples, our method is very simple and straightforward. DINOv2 adopts a dual-tower structure and adds a bunch of bells and whistles as shown in Table 1.

Furthermore, we would like to remind Reviewer aefc the very important comparison on vision and language tasks. Our method outperforms DINOv2 in 7 out of 9 tasks, and the overall score is significantly higher than that of DINOv2 (Dinov2 54.66 vs. Ours 58.94). Please note that the Dinov2 used here is distilled from the ViT-giant model. These improvements are significant.


We thank the reviewer bBvT for the comment, "At its core, this paper introduces a new way to pre-train open-world Vision-Language foundational models," and Reviewer W9bX for noting, "this is a new pretraining style for vision backbones rather than a variant of CLIP," our proposed new pretraining method is distinctly different from previous pretraining approaches. The differences from other most closely related classification-based pretraining methods are also evident: our method does not require manually curated "golden labels" and directly uses tokenized raw text as supervision. In the paper subsection "Word-level tokenizer vs. Subword-level tokenizer," we demonstrate through experiments that our method has better performance and scalability. As Reviewer 3a7k mentioned, "Compared to previous classification-based pretraining methods, the proposed approach demonstrates greater practical applicability." 

Overall, we believe we have provided a thorough response to Reviewer aefc's comments and hope that Reviewer aefc will review our feedback in light of the perspectives that align their rating with other reviewers.
Dear authors:

Thanks for your time, efforts, and response.

- **Q1** So far, we only see zero-shot performance on classification tasks. It seems that CLIP training style shows broader zero-shot ability in different downstream tasks.

- **Q2** From the comparison, I do not see superioty of the proposed pre-trained method compared to self-supervised learning especially the proposed method is actually weakly-supervised. Table 4 in DINOv2 provide a more comprehensive comparison.
It seems DINOv2 also have better transfer learning results in its Table 10.

Overall, I do not think the proposed methods show enough superiorty compared to CLIP and existing weakly-supervised/self-supervised methods, especially when there are some similar works like CatLIP.

At this time, I still hold my original score.

[1] https://arxiv.org/pdf/2304.07193
Dear Authors,

Thank you for providing the rebuttal response. 

While the proposed SuperClass method is not natively zero-shot, it shows other various flexibilities as demonstrated in the rebuttal. More importantly, this is a new pretraining style for vision backbones rather than a variant of CLIP. Honestly, the current pitch of the paper puts too much emphasis on CLIP (instead of advocating for vision backbone pretraining), and that is why many of the concerns from the reviewers are about comparisons with CLIP-like models. 

In the end, I believe this paper would allow the research community to improve vision backbone pretraining and have good insights. Therefore I will increase my score to weak accept, and hope that the paper is accepted. 

For the final version, I strongly recommend the authors to include all the rebuttal discussions in the main paper, and also put more emphasis on vision backbone pretraining, so that no ambiguities are left.
At its core, this paper introduces a new way to pre-train open-world Vision-Language foundational models, it's simple and elegant and seems to have maintained the zero-shot capabilities that CLIP offers. 

But I believe that using such a model might not be suited for generation, where one may want a text encoder to infer the relationships between words, i.e. "A dog over a cat" vs "A cat over a dog". 

Additionally, I believe that such a technique can also be used to clean the pertaining data of VLMs, a problem that LAION has faced in the past [1]. 

[1] https://cyber.fsi.stanford.edu/news/investigation-finds-ai-image-generation-models-trained-child-abuse

Thanks for the comprehensive answers to my questions, I will increase the score for soundness in my review.
Thank you for the response. I will maintain my score at this stage
>**Q4** Compared with these "bag-of-word classification" pre-trained methods

**A4.** We have included the comparison with other classification-based methods, like CatLIP in the subsection Word-level tokenizer vs. Subword-level tokenizer. The word-level tokenizer is used in CatLIP [46], which carefully selected approximately 40,000 "gold labels" from the datacomp-1B dataset. Aside from the tokenizer being different, all models are trained under the same settings. The results of Table 4 show that with the increasing size of the model, the subword-level tokenizer gradually outperforms the word-level tokenizer, whether in classification tasks or vision & language tasks.
We also provide the results of finetuning on ImageNet-1k in the below Table. The superclass could achieve better performance than CatLIP.
| Model | Pretraining |  ImageNet-1k Fine-tuning |
|---|---|---|
| OpenCLIP ViT-L/14 | Datacomp-1B | 87.4 |
| CatLIP ViT-L/16* | Datacomp-1B | 86.5 |
| Superclass ViT-L/16 | Datacomp-1B | 87.8 |
*number from the paper

>**Q5** GPU usage

**A5.** This is a typo. What we meant to say is all experiments were carried out on 80G A100 GPUs. We will fix it in the revised version.
> **Q1**. The effect of removing stop-words

In Table 7, removing stop-words leads to a decrease in classification accuracy, for example, the linear probing accuracy decreases from 76.0 to 75.7 (-0.3), and the zero-shot accuracy reduces from 61.7 to 61.0 (-0.7). Therefore, we conclude that stopwords could help the vision encoder. This might be explained by the fact that stop-words also carry some useful visual information. For example, 'she', 'her', and 'him' can indicate a person's gender; 'on', 'off', 'above', 'below', 'up', and 'down' can indicate operational status or position; '@' is likely to indicate an email address. 

Stop words and punctuation are listed below.
`['i', 'me', 'my', 'myself', 'we', 'our', 'ours', 'ourselves', 'you', "you're", "you've", "you'll", "you'd", 'your', 'yours', 'yourself', 'yourselves', 'he', 'him', 'his', 'himself', 'she', "she's", 'her', 'hers', 'herself', 'it', "it's", 'its', 'itself', 'they', 'them', 'their', 'theirs', 'themselves', 'what', 'which', 'who', 'whom', 'this', 'that', "that'll", 'these', 'those', 'am', 'is', 'are', 'was', 'were', 'be', 'been', 'being', 'have', 'has', 'had', 'having', 'do', 'does', 'did', 'doing', 'a', 'an', 'the', 'and', 'but', 'if', 'or', 'because', 'as', 'until', 'while', 'of', 'at', 'by', 'for', 'with', 'about', 'against', 'between', 'into', 'through', 'during', 'before', 'after', 'above', 'below', 'to', 'from', 'up', 'down', 'in', 'out', 'on', 'off', 'over', 'under', 'again', 'further', 'then', 'once', 'here', 'there', 'when', 'where', 'why', 'how', 'all', 'any', 'both', 'each', 'few', 'more', 'most', 'other', 'some', 'such', 'no', 'nor', 'not', 'only', 'own', 'same', 'so', 'than', 'too', 'very', 's', 't', 'can', 'will', 'just', 'don', "don't", 'should', "should've", 'now', 'd', 'll', 'm', 'o', 're', 've', 'y', 'ain', 'aren', "aren't", 'couldn', "couldn't", 'didn', "didn't", 'doesn', "doesn't", 'hadn', "hadn't", 'hasn', "hasn't", 'haven', "haven't", 'isn', "isn't", 'ma', 'mightn', "mightn't", 'mustn', "mustn't", 'needn', "needn't", 'shan', "shan't", 'shouldn', "shouldn't", 'wasn', "wasn't", 'weren', "weren't", 'won', "won't", 'wouldn', "wouldn't"]`

`{'{', '+', '(', '$', '}', '!', '%', '\\', '<', ';', '|', ']', '"', "'", ',', '&', '=', ')', '_', '^', '~', '#', '@', '.', '[', '*', '?', ':', '/', '>', '-'}`

> **Q2**. Online IDF

**A2**. To implement online IDF, we set up two global variables to track the number of seen samples (N) and the occurrence count of each subword. As training proceeds, these values will gradually approach the true word frequency of the entire dataset. The advantage of online IDF is its ease of use, allowing the code to be directly transferred to new datasets. We also conducted experiments to compare recognition accuracy using online IDF and offline IDF. The zero-shot accuracy with online IDF drops by about 0.3, and the linear probing accuracy drops by about 0.1.

> **Q3**. The analysis of classification results

**A3**. We evaluated the performance of CLIP and Superclass on the 1000 classes in ImageNet. Among them, Superclass performs better in 502 classes, while CLIP outperforms in 270 classes, and they perform equally well in the remaining classes. We also extracted the top 10 classes with the largest performance differences and displayed them in the table below. We do not find any obvious patterns, but it is possible that Superclass and CLIP can complement each other. This is an interesting direction for future research.

| Class name | muzzle | music speaker | yellow garden spider | cardboard box / carton | Carolina anole | black-footed ferret | monitor | missile | cricket insect | garter snake |
|------------|--------|---------------|----------------------|------------------------|----------------|---------------------|---------|---------|----------------|--------------|
| CLIP       | 0.72   | 0.68          | 0.82                 | 0.66                   | 0.66           | 0.62                | 0.36    | 0.44    | 0.58           | 0.82         |
| SuperClass | 0.88   | 0.84          | 0.96                 | 0.80                   | 0.81           | 0.76                | 0.50    | 0.58    | 0.70           | 0.94         |

| Class name | European polecat | oxygen mask | parallel bars | ox   | promontory | English Setter | stethoscope | split-rail fence | rotisserie | cassette player |
|------------|------------------|-------------|---------------|------|------------|----------------|-------------|------------------|------------|-----------------|
| CLIP       | 0.64             | 0.78        | 0.82          | 0.68 | 0.62       | 0.92           | 0.94        | 0.88             | 0.98       | 0.48            |
| SuperClass | 0.44             | 0.64        | 0.70          | 0.58 | 0.52       | 0.82           | 0.84        | 0.78             | 0.88       | 0.38            |