FastCLIP: A Suite of Optimization Techniques to Accelerate CLIP Training with Limited Resources

Authors: Xiyuan Wei, Fanjiang Ye, Ori Yonay, Xingyu Chen, Baixi Sun, Dingwen Tao, Tianbao Yang
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Multimodal Learning, Contrastive Learning

Abstract:
Existing studies of training state-of-the-art Contrastive Language-Image Pretraining (CLIP) models on large-scale data involve hundreds of or even thousands of GPUs due to the requirement of a large batch size. However, such a large amount of resources is not accessible to most people. While advanced compositional optimization techniques for optimizing global contrastive losses have been demonstrated effective for removing the requirement of a large batch size, their performance on large-scale data remains underexplored and not optimized. To bridge the gap, this paper explores several aspects of CLIP training with \textit{limited resources} (e.g., up to tens of GPUs). First, we introduce FastCLIP, a general CLIP training framework built on advanced compositional optimization techniques while designed and optimized for the {\bf distributed setting}. Our framework is equipped with an efficient gradient reduction strategy to reduce communication overhead. Second, to further boost training efficiency, we investigate three components of the framework from an optimization perspective: the schedule of the inner learning rate, the update rules of the temperature parameter and the model parameters, respectively. Experiments on different strategies for each component shed light on how to conduct CLIP training more efficiently. Finally, we evaluate the performance of FastCLIP and the state-of-the-art training baseline (OpenCLIP) on different compute scales up to 32 GPUs on 8 nodes, and three data scales ranging from 2.7 million, 9.1 million to 315 million image-text pairs to demonstrate the significant improvement of FastCLIP in the resource-limited setting.

Decision: reject
Reviewer scores: 3, 5, 3

Reviews and discussion:
<thread>
creation date:
1730240038535

thread_type:
official_review

<thread_object>
creator: reviewer_rd79

<content>
summary:
This paper introduced FastCLIP, which uses data-parallel for CLIP training. It claimed to (1) present an efficient gradient reduction strategy to reduce communication overhead, (2) compare inner LR schedule between constant and cosine schedule, (3) compare temperature param update methods and (4) compare optimizers.

strengths:
1. This paper compared CLIP training performances with different temperature parameter updating methods and different optimizers.
2. The appendix provides detailed experiment results and parameter settings.
3. An introduced cosine schedule could provide better training accuracy.

weaknesses:
1. Figure 3 is not so clear about the efficiency improvement. The difference between OpenCLIP and FastCLIP is not significant.
2. Figure 4 (a) does not show convergence on the accuracy curve which could be problematic.
3. Figure 4 (b) (c) do not show a significant difference in scalability.
4. The writing style needs improvement, and some part of this paper needs to be summarized. Also, the experiment part needs more details, the authors use many numbers in Table 3, 4, 5 but there is no information on what do they represent, accuracy, or some other metrics.
5. Lack of novelty. This paper compared multiple combinations of existing CLIP training techniques and proposed using data parallelization to scale up the training process. From this summary, existing techniques do not contribute much to novelty except cosine LR schedule, also, data parallelism mentioned in this paper does not show promising results compared to OpenCLIP.

questions:
1. See weaknesses.
2. Is there any other significant difference between OpenCLIP and FastCLIP? Seems like they are also using data-parallel.
3. What do the number differences represent in Table 3, 4, 5, and Figure 2? The current comparison is not very straightforward.
4. Should the training accuracy converge?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730370175202

thread_type:
official_review

<thread_object>
creator: reviewer_rov8

<content>
summary:
This paper introduces a distributed framework designed to improve the efficiency of CLIP by optimizing resource use and reducing dependency on large batch sizes and numerous GPUs. They use several optimization techniques, including a gradient reduction strategy and a flexible LR schedule, to facilitate efficient CLIP training with limited resources. The study compares FastCLIP to OpenCLIP and demonstrates consistent performance improvements on different scales of data and compute resources.

strengths:
- Efficient resource utilization for settings with limited computational resources, making CLIP training more accessible.

- Systematic Optimizations: The paper provides a structured approach to optimizing multiple aspects of CLIP training, including LR schedules, temperature parameter updates, and gradient communication strategies.

- Experimental Validation: Comprehensive testing on various datasets and compute scales effectively shows the benefits of FastCLIP compared to OpenCLIP.

weaknesses:
- Limited Novelty in Optimization Techniques: The optimization techniques applied, such as learning rate decay and gradient reduction, are well-established and may not contribute novel methodological insights.

- Resource Comparison Limitations: The paper does not conduct an extensive ablation study for larger datasets due to resource constraints, which may limit the generalizability of results on extremely large data scales.

- Assumption of Availability of Multiple GPUs: Although designed for resource-limited environments, FastCLIP still requires access to multiple GPUs, which may limit applicability for highly resource-constrained users.

questions:
How does the performance of FastCLIP vary across different types of GPUs or compute environments?

Could other advanced optimization techniques (e.g., adaptive optimizers beyond AdamW) further enhance the framework's efficiency?

Would additional experiments on extremely large datasets (>1 billion samples) align with the current findings for smaller datasets?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730707486300

thread_type:
official_review

<thread_object>
creator: reviewer_5gwx

<content>
summary:
This paper introduces FastCLIP, a distributed training framework designed to optimize CLIP model training using compositional optimization techniques, removing the dependency on large batch sizes for effective model performance. The framework incorporates an efficient gradient reduction strategy to minimize communication overhead and conducts comprehensive ablation studies on various components, including learning rate schedules (constant vs. cosine), temperature parameter update rules, and different optimizers (AdamW, LAMB, Lion, and SGDM).

strengths:
1. This paper studies an important problem in CLIP training, "how to efficiently and effectively training CLIP models with limited resources". 

2. This paper designs an efficient distributed training framework based on advanced compositional optimization techniques. It conducts ablation studies on the several key components during training, such as the update rule of learning rate, temperature parameter and model papers, providing valuable insights for future work in optimizing large-scale model training.

3. Experimental results across various compute and data scales demonstrate that FastCLIP significantly outperforms existing methods, such as OpenCLIP, enhancing training efficiency on setups with up to 32 GPUs.

weaknesses:
1. This paper is not designed for general CLIP training but is instead built on the assumptions and techniques from prior work[1]. As a result, its applicability may be limited to scenarios that align with these specific assumptions, restricting its generalizability to broader CLIP training tasks.

2. The paper primarily builds on existing techniques, such as compositional optimization and gradient reduction strategies, without introducing fundamentally new concepts. While it refines and optimizes these methods for CLIP training, the core ideas themselves are not particularly novel.

3. The paper's writing lacks clarity, making it difficult to follow at times. The structure of the paper would benefit from significant revision to improve its readability and logical consistency.

4. The experiment compares FastCLIP with OpenCLIP, which is designed for large-scale clusters and requires large batch sizes. In contrast, FastCLIP is tested on a relatively smaller cluster (up to 32 GPUs), violating the original conditions under which OpenCLIP was evaluated. Additionally, the results in experiments 4 and 5 are inconsistent, with different methods showing unstable performance. This lack of stability weakens the conclusions and limits the insights that can be drawn for future work.

[1] Yuan Z, Wu Y, Qiu Z H, et al. Provable stochastic optimization for global contrastive learning: Small batch does not harm performance. ICML 2022.

questions:
See the weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732405056383

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to Common Questions Raise by Reviewers

comment:
We sincerely thank all reviewers for their valuable review. Here we address common questions raised by the reviewers, then we address each reviewer's questions in their individual rebuttal.

**Q1**: Novelty of the FastCLIP framework.

*A1*: The key novelty of FastCLIP lies at the **distributed infrastructure** to support algorithms that optimize **Global Contrastive Loss** of contrastive learning. FastCLIP is different from existing open-soured distributed frameworks for CLIP training, e.g., OpenCLIP. In particular, OpenCLIP supports algorithms based on mini-batch contrastive losses. However, these algorithms require a large batch size to work well. Recent advances have proposed efficient algorithms for optimizing global contrastive losses, including SogCLR and iSogCLR. But there is lack of distributed infrastructure to support them for training on billions of data points.  In this distributed infrastructure, 
- we proposed a novel gradient reduction strategy, which reduces the communication costs of OpenCLIP. 
- we designed better optimization strategies for different components within the framework.

**Q2**: Writing and exposition of the paper.

*A2*: This paper is presented in the following manner. First we introduce the overall design and the gradient reduction strategy of the FastCLIP framework. Next, we conduct extensive experiments comparing new strategies for different components within the framework with existing strategies. In the end, we demonstrate the scaling performance of our most performant algorithm FastCLIP-v3 and compare it with OpenCLIP at different scales to show the effectiveness of FastCLIP in limited-resource settings. We added a paragraph in the revision describing the exposition at the end of Introduction to help readers follow the paper.

**Q3**: Results of FastCLIP-v3 on extremely large datasets (>1 billion examples).

*A3*: We are conducting an experiment of FastCLIP-v3 on a 1.4B subset of the DFN-2B dataset (Fang et al. 2023). The number of samples we finished at this moment is 11.2B. We compare the performance of FastCLIP-v3 on this data to that trained on LAION-315M reported in the paper in the following table, where the ImageNet-1K top1 accuracy at different number of samples seen are presented. We can see that our approach is still effective on extremely large-scale data. We also plot the performance curves of FastCLIP-v3 on different datasets and present it in Appendix E.2 of the revision. We expect to finish the training with 12.8B samples seen and add the result in the next revision.

| Work | Data | Acc. (2.8B) | 5.6B | 8.4B | 11.2B |
| --- | --- | --- | --- | --- | --- |
| Ours | DFN-1.4B | 53.76 | 57.16 | 61.81 | 70.3 |
| Ours | LAION-315M | 47.01 | 51.52 | 56.56 | 60.1 |
</content>
</thread_object>

</thread>


Metareview:
metareview:
Thank you for your submission to ICLR. This paper introduces FastCLIP, a set of techniques and implementation for efficient training of CLIP in settings with limited resources. To carry this out, the authors developed a gradient reduction strategy to reduce communication requirements, and then investigated aspects of the optimization procedure (e.g., inner learning rate schedule, update rule of the temperature/model parameters).

The reviewers agree with the motivation for this paper, and all appreciate the thorough set of empirical results and ablations. However, the reviewers also had a number concerns, including:
- Limited novelty of the proposed techniques.
- In some cases, there was not a great deal of demonstrated improvement of the proposed method over existing baselines.
- Issues with clarity in the writing and in certain figures.

I encourage you to carefully consider and incorporate these comments from reviewers upon resubmission.