Enhancing Vision-Language Model Pre-training with Image-text Pair Pruning Based on Word Frequency

Authors: Mingliang Liang, Martha Larson
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Vision-Language Model, Multimodal Data, Data Pruning

Abstract:
We propose Word-Frequency-based Image-Text Pair Pruning (WFPP), a novel data pruning method that improves the efficiency of VLMs.
Unlike MetaCLIP, our method does not need metadata for pruning, but selects text-image pairs to prune based on the content of the text. Specifically, WFPP prunes text-image pairs containing high-frequency words across the entire training dataset. The effect of WFPP is to reduce the dominance of frequent words. The result a better balanced word-frequency distribution in the dataset, which is known to improve the training of word embedding models. After pre-training on the pruned subset, we fine-tuned the model on the entire dataset for one additional epoch to achieve better performance. Our experiments demonstrate that applying WFPP when training a CLIP model improves performance on a wide range of downstream tasks. WFPP also provides the advantage of speeding up pre-training by using fewer samples. Additionally, we analyze the training data before and after pruning to visualize how WFPP changes the balance of word frequencies. We hope our work encourages researchers to consider the distribution of words in the training data when pre-training VLMs, not limited to CLIP.

Decision: reject
Reviewer scores: 3, 8, 8, 3

Reviews and discussion:
<thread>
creation date:
1730295536596

thread_type:
official_review

<thread_object>
creator: reviewer_zmle

<content>
summary:
This paper introduces a data pruning method for CLIP that focuses on removing frequently occurring data samples. The experimental results show that the pruned model outperforms the original CLIP across several datasets and downstream tasks, indicating that eliminating high-frequency samples can enhance CLIP's training efficiency and performance.

strengths:
Simplicity: The proposed approach is intuitive and easy to implement, which enhances its potential for broader application.

Performance Gains: The method consistently outperforms the original CLIP across most datasets, demonstrating that the removal of high-frequency samples leads to improved model performance.

weaknesses:
Insufficient Coverage of Related Work: The paper lacks citations and discussions of key related works in data pruning. Crucial references such as [1-3] are missing. Including these would better position the contribution within the current research landscape.

Limited Baseline Comparisons: The experimental comparison is restricted to the CLIP and MetaCLIP, neglecting other state-of-the-art (SOTA) data pruning methods. A broader comparison with cutting-edge approaches would strengthen the evaluation and provide a more comprehensive assessment of the method's effectiveness.

Narrow Model Scope: The pruning method is evaluated only on CLIP, without experiments on other architectures such as BLIP or Llava, which are also highly relevant for vision-language and generative tasks. Additionally, the paper does not explore how the proposed method performs on more challenging vision-language tasks like VQA or image captioning. Expanding the experimental scope to include both different models and more challenging tasks would help to establish the method’s generalizability and effectiveness across diverse architectures and applications.

[1] Beyond Neural Scaling Laws: Beating Power Law Scaling via Data Pruning.

[2] When Less is More: Investigating Data Pruning for Pretraining LLMs at Scale.

[3] Data Pruning via Moving-one-Sample-out.

questions:
Please see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730625604926

thread_type:
official_review

<thread_object>
creator: reviewer_ele2

<content>
summary:
This paper introduces Word-Frequency-based Image-Text Pair Pruning (WFPP), a novel approach designed to enhance the selection of image-text pairs for more effective CLIP pre-training.

strengths:
1. The paper is clearly written and structured in a way that is easy to understand.

2. The experiments conducted on robustness classification tasks are thoroughly executed.

weaknesses:
1. While CLIP is over three years old and attention has shifted to newer multimodal large language models (MLLMs), the impact of WFPP on larger models such as LLaVA remains under-explored.
2. MetaCLIP [1] has demonstrated impressive success in establishing a comprehensive pipeline for entire CLIP pre-training. However, the WFPP experiments are conducted using CC3M and CC12M datasets, which are significantly smaller than the 400M images used in the original CLIP training. This raises questions about the scalability and usefulness of WFPP when applied to larger datasets.
3. Although the authors claim that WFPP can enhance performance across a broad range of downstream tasks, the comparison baselines only include the original CLIP, which may not sufficiently demonstrate the method's relative effectiveness.
4. The proposed method employs techniques that are simple and already widely used in the NLP community. Therefore, more comprehensive experiments and detailed discussions are crucial to substantiate its uniqueness and effectiveness.

[1] Demystifying clip data. ICLR 2024

questions:
1. Is there any analysis or discussion in the paper that supports the effectiveness of WFPP when applied to much larger datasets, such as LAION?

2. It would be beneficial for the authors to compare WFPP with improved CLIP pre-training baselines, such as DeCLIP [2], FILIP [3], and UniCLIP [4], to better showcase the strengths of their approach.

3. Given that other works, such as TinyCLIP [5] and MoPE-CLIP [6], have achieved competitive performance by pruning and training CLIP models on datasets like CC12M or YFCC15M, a discussion and comparison with these studies would provide a more comprehensive view of WFPP's comparative advantages.

4. Numerous data processing techniques have been proposed to enhance CLIP pre-training, such as DataCompDR in MobileCLIP [7] and the OFA module in ALIP [8]. How does WFPP compare to these strategies in terms of improving pre-training performance?

[2] Supervision exists everywhere: A data-efficient contrastive language-image pre-training paradigm. ICLR 2022

[3] Filip: fine-grained interactive language-image pre-training. ICLR 2022

[4]  Uniclip: Unified framework for contrastive language-image pretraining. NeurIPS 2022

[5] Tinyclip: Clip distillation via affinity mimicking and weight inheritance. ICCV 2023

[6] Mope-clip: Structured pruning for efficient vision-language models with module-wise pruning error metric. CVPR 2024

[7] Mobileclip: Fast image-text models through multi-modal reinforced training. CVPR 2024

[8] Alip: Adaptive language-image pre-training with synthetic caption. ICCV 2023
</content>
</thread_object>

</thread>
<thread>
creation date:
1730729192906

thread_type:
official_review

<thread_object>
creator: reviewer_vhze

<content>
summary:
This paper hypothesizes that a balanced word-frequency distribution in a dataset can improve the training efficiency of Visual Language Models (VLMs). Through extensive experiments, the authors demonstrate that their proposed method, Word-Frequency-based Image-Text Pair Pruning (WFPP), achieves better performance with a reduced training budget.

strengths:
1. The approach is straightforward and constructive.
2. The authors conduct comprehensive experiments to validate WFPP’s effectiveness.
3. The analysis on text length normalization provides valuable insights.
4. The authors make all code and data publicly available, enhancing reproducibility.

weaknesses:
1. The paper lacks theoretical analysis explaining why pruning based on word frequency is effective for image-text alignment.

questions:
1. What does “w/o ft WFPP” mean in Figure 1?
2. Existing methods emphasize the importance of re-captioning. Would the proposed method be affected by re-captioning? What if, instead of deleting image-text pairs, we re-captioned the images?
3. WFPP uses word frequency as an indicator. Have the authors considered using alternative criteria, such as n-grams or token frequency?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730732972793

thread_type:
official_review

<thread_object>
creator: reviewer_trgd

<content>
summary:
This paper introduces Word-Frequency-based Image-Text Pair Pruning (WFPP), a method to enhance the pretraining of Vision-Language Models (VLMs) by pruning the training dataset based on word frequency. The authors argue that reducing the dominance of high-frequency words leads to a better-balanced word-frequency distribution, which in turn improves the training of word embedding models. The experiments show that their approach, when applied to training a CLIP model, improves performance on various downstream tasks and speeds up pre-training by using fewer samples.

strengths:
- The proposed method is simple yet effective, and the authors demonstrate its superiority on various benchmarks over other pruning methods for both pre-training and fine-tuning. 

- The paper is well-written, and the figures and tables are clear and support the narrative effectively.

weaknesses:
- The proposed method is verified on small scale dataset (CC3M and CC12M) with a large gap of accuracy against existsing methods (e.g., CLIP, MetaCLIP.) pretrained on larger scales (i.e., CommonCrawl 400M). For instance, the zero-shot classification accuracy on ImageNet 1K is around 35%+ for WFPP, but CLIP and MetaCLIP enjoys around 70%. Thus it may still remains unsure if WFPP can be extent to larger scales, expecially as a method to prune the large scale pre-training dataset.

- The probability to discard a sample decreases with the increasing length of input text, and this holds for both Equation 3 and Equation 4. This is intuitively unreasonable since longer input would be less likely to be discarded. The text length is set to 32 by default, but in practice the text lenght can vary a lot and more experiments w.r.t. the text length should be discussed.

questions:
- The paper uses a straightforward approach to calculate word frequency, which is essential for the WFPP method. It would be beneficial if the authors could discuss the choice of this particular frequency calculation over other potential methods, such as term frequency-inverse document frequency (TF-IDF) or others.

- The threshold $t$ seems to be a critical hyper-parameter. The default value 1e-7 seems to be tricky. The authors should provide more details on how t was chosen and whether its value was tuned experimentally or derived theoretically. Additionally, discussing the sensitivity of the model's performance to changes in t would be insightful.

- It would be helpful to visualize the words filtered, and change of data category proportion before and after WFPP.

- How about the potential of WFPP on (vision-text) large language models?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732570119227

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear reviewers and ACs,

Thank you for your thorough review of our submission. We greatly appreciate your constructive feedback and would like to address the weaknesses you've identified.

Strengths:

 We are pleased that the reviewers appreciate our method's simplicity and practicality (**Reviewer** **TRgD,  VHZe, eLe2, zmLE**). The reviewers have also recognized that it consistently outperforms existing models and other pruning methods across various benchmarks through comprehensive experiments (**Reviewer** **TRgD,  VHZe, eLe2)**. Further, the reviewers comment that the paper is well-written, clearly structured, and includes helpful visual aids like figures and tables, making the content accessible and easy to follow (**Reviewer** **TRgD,  VHZe, zmLE**). We have made all code and data publicly available to ensure reproducibility.

Weaknesses:

The CLIP model that we used in our experiments was pre-trained on CC3M and CC12M datasets, and does not achieve the same performance of CLIP models trained on larger data sets, such as the original CLIP[8] and MetaCLIP [1] (**Reviewer** **TRgD, eLe2**). Due to limited computing resources, we cannot train CLIP on large-scale datasets like LAION-400M. Based on WFPP's successful application in CLIP with the CC3M and CC12M datasets, we believe that WFPP can also apply on large-scale datasets. 

Additionally, the paper needs more comprehensive citations and discussion of related work in efficient CLIP training and data pruning (**Reviewer eLe2, zmLE**).  We've added your recommended related works to cover different efficient CLIP training methods and data pruning approaches in NLP and CV. 

While our experimental comparisons focus primarily on the original CLIP and MetaCLIP models, our data pruning method is complementary to other efficient CLIP training methods, such as DeCLIP [2], Filip [3], Uniclip [4], TinyCLIP [5], MobileCLIP [6], and ALIP [7] (**Reviewer eLe2)**. WFPP can be applied to these approaches to further improve their training efficiency.  Given the time limit of rebutal, It’s impossible to evaluate our method on them one by one. Instead, we provide an analysis on the additional datasets, COCO and VG, used in other method, where we show that the word distribution after applying WFPP is similar to the word distribution CC12M, which prune the text that inculde high frequency words to create a balance subset. Experimental results on additional data sets are shown in Figure 4,5,6 and 7 in our revised paper.  

Moreover, we made this additions to the revised paper that we have uploaded in response to the reviewer comments: We conducted additional experiments to demonstrate our method's effectiveness with synthetic captions (**Question from** **Reviewer VHZe**). We analyzed threshold selection (**Question from** **Reviewer TRgD**) and performed detailed word and token analysis (**Question from Reviewer VHZe**). We also carried out data analysis to show the potential for our method can be applied across different datasets and models (**Question from** **Reviewer TRgD**). 

We trust this response addresses your concerns and clarifies our work. We appreciate your feedback and look forward to further improving our paper based on your suggestions.

Best regards,

Authors

[1]  Demystifying clip data. ICLR 2024

[2] Supervision exists everywhere: A data-efficient contrastive language-image pre-training paradigm. ICLR 2022

[3] Filip: fine-grained interactive language-image pre-training. ICLR 2022

[4] Uniclip: Unified framework for contrastive language-image pretraining. NeurIPS 2022

[5] Tinyclip: Clip distillation via affinity mimicking and weight inheritance. ICCV 2023

[6] Mobileclip: Fast image-text models through multi-modal reinforced training. CVPR 2024

[7] Alip: Adaptive language-image pre-training with synthetic caption. ICCV 2023

[8] Learning Transferable Visual Models From Natural Language Supervision. ICML 2021
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes WFPP, a word-frequency-based pruning method for improving CLIP pre-training by removing image-text pairs containing high-frequency words. While the paper demonstrates some empirical improvements on CC3M/CC12M datasets and offers a simple, practical approach, several critical weaknesses make it unsuitable for acceptance. The key limitations include: (1) All experiments are conducted only on small datasets (CC3M/CC12M) with performance far below state-of-the-art models trained on larger datasets like LAION-400M, raising serious questions about scalability. (2) The theoretical foundation is weak - there's no clear explanation for why pruning based on word frequency should help with image-text alignment. (3) The method has concerning technical issues, like discarding probability being dependent on text length in an unintuitive way. (4) The experimental evaluation is limited - comparisons focus mainly on basic CLIP rather than more recent efficient training methods. Though the authors added some new experiments during rebuttal (e.g., with DeCLIP and synthetic captions), fundamental questions about scalability to large datasets remain unaddressed. Thus I vote to reject the paper.