File size: 6,855 Bytes

---
language:
- en
---
# VeCLIP: Improving CLIP Training via Visual-enriched Captions

* A novel CLIP training scheme that achieves the SoTA performance on zero-shot ImageNet classification and COCO image text retreival using limited visual-enriched captions.* [[Paper](https://arxiv.org/abs/2310.07699)]

[Zhengfeng Lai*](https://zjujefflai.github.io/), [Haotian Zhang*](https://haotian-zhang.github.io/) , [Bowen Zhang](https://zbwglory.github.io/), Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, [Zhe Gan](https://zhegan27.github.io/), Jiulong Shan, [Chen-Nee Chuah](https://www.ece.ucdavis.edu/~chuah/rubinet/people/chuah/bio.html), Yinfei Yang, Meng Cao [*: equal contribution]


<p align="center">
    <img src="veclip_diagram.jpg" width="100%"></a> <br>
    Diagram of VeCap.
</p>

## Release
- [03/06/2024] 🔥 We released the VeCLIP & VeCap-DFN [checkpoints](#checkpoints).

## Contents
- [Install](#install)
- [Getting Started](#getting-started)
- [Checkpoints](#checkpoints)

## Install

1. Clone this repository
```Shell
git clone https://github.com/apple/ml-veclip
cd ml-veclip
```

2. Create an environment and install related packages
```Shell
conda create -n veclip python=3.9 -y
conda activate veclip
pip install -r requirements.txt
```

## Getting Started

See the [example notebook](https://github.com/apple/ml-veclip/blob/main/load_veclip.ipynb) for details on how to simply load the different checkpoints using HuggingFace transformers.


## Checkpoints

We release the checkpoints for **VeCLIP**, which are trained from scratch on visual-enriched captions VeCap 3M/12M/100M/200M, as reported in the paper. The models are evaluated on COCO/Flickr30k image-text retrieval and ImageNet/ImageNetv2 classification in a zero-shot fashion. Use `wget` or `curl` to download the below checkpoints. 

<table>
<thead>
  <tr>
    <th rowspan="2">Data</th>
    <th rowspan="2">Model</th>
    <th rowspan="2">Resolution</th>
    <th colspan="2">COCO (R@1)</th>
    <th colspan="2">Flickr30k (R@1)</th>
    <th rowspan="2">ImageNet</th>
    <th rowspan="2">ImageNetv2</th>
  </tr>
  <tr>
    <th>I2T</th>
    <th>T2I</th>
    <th>I2T</th>
    <th>T2I</th>
  </tr>
</thead>
<tbody>
  <tr>
    <td rowspan="2">VeCap 3M</td>
    <td>CLIP-B/16</td>
    <td>224x224</td>
    <td>5.46</td>
    <td>3.28</td>
    <td>12.20</td>
    <td>6.36</td>
    <td>5.46</td>
    <td>7.09</td>
  </tr>
  <tr>
    <td><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/veclip_b16_3m.zip">VeCLIP-B/16</a></td>
    <td>224x224</td>
    <td>22.30</td>
    <td>13.01</td>
    <td>40.60</td>
    <td>27.58</td>
    <td>15.98</td>
    <td>13.51</td>
  </tr>
  <tr>
    <td rowspan="2">VeCap 12M</td>
    <td>CLIP-B/16</td>
    <td>224x224</td>
    <td>24.52</td>
    <td>14.28</td>
    <td>44.70</td>
    <td>290.6</td>
    <td>31.60</td>
    <td>27.03</td>
  </tr>
  <tr>
    <td><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/veclip_b16_12m.zip">VeCLIP-B/16</a></td>
    <td>224x224</td>
    <td>47.78</td>
    <td>31.62</td>
    <td>73.90</td>
    <td>55.68</td>
    <td>38.11</td>
    <td>32.53</td>
  </tr>
  <tr>
    <td rowspan="2">VeCap 100M</td>
    <td>CLIP-B/16</td>
    <td>224x224</td>
    <td>47.24</td>
    <td>30.61</td>
    <td>74.40</td>
    <td>57.16</td>
    <td>58.64</td>
    <td>50.96</td>
  </tr>
  <tr>
    <td><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/veclip_b16_100m.zip">VeCLIP-B/16</a></td>
    <td>224x224</td>
    <td>64.82</td>
    <td>46.12</td>
    <td>89.30</td>
    <td>73.10</td>
    <td>60.77</td>
    <td>54.17</td>
  </tr>
  <tr>
    <td rowspan="2">VeCap 200M</td>
    <td>CLIP-B/16</td>
    <td>224x224</td>
    <td>52.20</td>
    <td>34.97</td>
    <td>80.90</td>
    <td>63.26</td>
    <td>63.72</td>
    <td>56.84</td>
  </tr>
  <tr>
    <td><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/veclip_b16_200m.zip">VeCLIP-B/16</a></td>
    <td>224x224</td>
    <td>67.20</td>
    <td>48.40</td>
    <td>91.10</td>
    <td>76.32</td>
    <td>64.64</td>
    <td>57.67</td>
  </tr>
</tbody>
</table>


We further found our VeCap can also be complementary to other well-established filtering methods, e.g., [Data Filtering Network (DFN)](https://arxiv.org/abs/2309.17425). We also provide thosse checkpoints (referred to as **VeCap-DFN**) and report their performance below. 

<table>
<thead>
<tr>
<th rowspan="2">Backbone</th>
<th rowspan="2">Resolution</th>
<th rowspan="2">Data</th>
<th colspan="2">COCO (R@1)</th>
<th colspan="2">Flickr30k (R@1)</th>
<th rowspan="2">ImageNet</th>
<th rowspan="2">ImageNetV2</th>
</tr>
<tr>
<th>I2T</th>
<th>T2I</th>
<th>I2T</th>
<th>T2I</th>

</tr>
</thead>
<tbody>
<tr>
<td rowspan="3"><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/vecapdfn_clip_b16.zip">VeCap-DFN-B/16</a></td>
<td rowspan="3">224x224</td>
<td>DFN </td>
<td>62.96</td>
<td>43.20</td>
<td>87.10</td>
<td>70.44</td>
<td>76.15</td>
<td>68.19</td>
</tr>
<tr>
<td>VeCap 300M</td>
<td>64.74</td>
<td>44.58</td>
<td>90.10</td>
<td>73.14</td>
<td>46.43</td>
<td>41.15</td>
</tr>
<tr>
<td>DFN + VeCap 300M</td>
<td>66.28</td>
<td>45.12</td>
<td>88.80</td>
<td>73.56</td>
<td>76.19</td>
<td>69.58</td>
</tr>
<tr>
<td rowspan="1"><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/vecapdfn_clip_l14.zip">VeCap-DFN-L/14</a></td>
<td rowspan="1">224x224</td>
<td>DFN + VeCap 300M</td>
<td>71.06</td>
<td>51.13</td>
<td>93.10</td>
<td>80.96</td>
<td>81.95</td>
<td>75.48</td>
</tr>
<tr>
<td rowspan="2"><a href="https://docs-assets.developer.apple.com/ml-research/models/veclip/vecapdfn_clip_h14_336.zip">VeCap-DFN-H/14</a></td>
<td rowspan="1">336x336</td>
<td>DFN + VeCap 300M</td>
<td>72.78</td>
<td>52.33</td>
<td>93.60</td>
<td>82.64</td>
<td>83.07</td>
<td>76.37</td>
</tr>
</tbody>
</table>


## Citation

If you find VeCLIP useful, please cite using this BibTeX:

```bibtex
@article{lai2023scarcity,
  title={From scarcity to efficiency: Improving clip training via visual-enriched captions},
  author={Lai, Zhengfeng and Zhang, Haotian and Zhang, Bowen and Wu, Wentao and Bai, Haoping and Timofeev, Aleksei and Du, Xianzhi and Gan, Zhe and Shan, Jiulong and Chuah, Chen-Nee and Yang, Yinfei and others},
  journal={arXiv preprint arXiv:2310.07699},
  year={2023}
}
@article{fang2023data,
  title={Data filtering networks},
  author={Fang, Alex and Jose, Albin Madappally and Jain, Amit and Schmidt, Ludwig and Toshev, Alexander and Shankar, Vaishaal},
  journal={arXiv preprint arXiv:2309.17425},
  year={2023}
}
```

## Acknowledgement

- [axlearn](https://github.com/apple/axlearn): the codebase we use to train the models. 
- [huggingface transformers](https://huggingface.co/docs/transformers/en/index): Transformers provides APIs to load our trained models.