|
--- |
|
license: apache-2.0 |
|
--- |
|
|
|
|
|
# Chinese-CLIP-ViT-Base-Patch16 |
|
|
|
## Introduction |
|
This is the base-version of the Chinese CLIP, with ViT-B/16 as the image encoder and RoBERTa-wwm-base as the text encoder. Chinese CLIP is a simple implementation of CLIP on a large-scale dataset of around 200 million Chinese image-text pairs. For more details, please refer to our technical report https://arxiv.org/abs/2211.01335 and our official github repo https://github.com/OFA-Sys/Chinese-CLIP |
|
|
|
## Use with the official API |
|
We provide a simple code snippet to show how to use the API for Chinese-CLIP. For starters, please install cn_clip: |
|
```bash |
|
# to install the latest stable release |
|
pip install cn_clip |
|
|
|
# or install from source code |
|
cd Chinese-CLIP |
|
pip install -e . |
|
``` |
|
After installation, use Chinese CLIP as shown below: |
|
```python |
|
import torch |
|
from PIL import Image |
|
|
|
import cn_clip.clip as clip |
|
from cn_clip.clip import load_from_name, available_models |
|
print("Available models:", available_models()) |
|
# Available models: ['ViT-B-16', 'ViT-L-14', 'ViT-L-14-336', 'ViT-H-14', 'RN50'] |
|
|
|
device = "cuda" if torch.cuda.is_available() else "cpu" |
|
model, preprocess = load_from_name("ViT-B-16", device=device, download_root='./') |
|
model.eval() |
|
image = preprocess(Image.open("examples/pokemon.jpeg")).unsqueeze(0).to(device) |
|
text = clip.tokenize(["杰尼龟", "妙蛙种子", "小火龙", "皮卡丘"]).to(device) |
|
|
|
with torch.no_grad(): |
|
image_features = model.encode_image(image) |
|
text_features = model.encode_text(text) |
|
# Normalize the features. Please use the normalized features for downstream tasks. |
|
image_features /= image_features.norm(dim=-1, keepdim=True) |
|
text_features /= text_features.norm(dim=-1, keepdim=True) |
|
|
|
logits_per_image, logits_per_text = model.get_similarity(image, text) |
|
probs = logits_per_image.softmax(dim=-1).cpu().numpy() |
|
|
|
print("Label probs:", probs) # [[1.268734e-03 5.436878e-02 6.795761e-04 9.436829e-01]] |
|
``` |
|
|
|
However, if you are not satisfied with only using the API, feel free to check our github repo https://github.com/OFA-Sys/Chinese-CLIP for more details about training and inference. |
|
<br><br> |
|
|
|
## Results |
|
**MUGE Text-to-Image Retrieval**: |
|
<table border="1" width="100%"> |
|
<tr align="center"> |
|
<th>Setup</th><th colspan="4">Zero-shot</th><th colspan="4">Finetune</th> |
|
</tr> |
|
<tr align="center"> |
|
<td>Metric</td><td>R@1</td><td>R@5</td><td>R@10</td><td>MR</td><td>R@1</td><td>R@5</td><td>R@10</td><td>MR</td> |
|
</tr> |
|
<tr align="center"> |
|
<td width="120%">Wukong</td><td>42.7</td><td>69.0</td><td>78.0</td><td>63.2</td><td>52.7</td><td>77.9</td><td>85.6</td><td>72.1</td> |
|
</tr> |
|
<tr align="center"> |
|
<td width="120%">R2D2</td><td>49.5</td><td>75.7</td><td>83.2</td><td>69.5</td><td>60.1</td><td>82.9</td><td>89.4</td><td>77.5</td> |
|
</tr> |
|
<tr align="center"> |
|
<td width="120%">CN-CLIP</td><td>63.0</td><td>84.1</td><td>89.2</td><td>78.8</td><td>68.9</td><td>88.7</td><td>93.1</td><td>83.6</td> |
|
</tr> |
|
</table> |
|
<br> |
|
|
|
**Flickr30K-CN Retrieval**: |
|
<table border="1" width="120%"> |
|
<tr align="center"> |
|
<th>Task</th><th colspan="6">Text-to-Image</th><th colspan="6">Image-to-Text</th> |
|
</tr> |
|
<tr align="center"> |
|
<th>Setup</th><th colspan="3">Zero-shot</th><th colspan="3">Finetune</th><th colspan="3">Zero-shot</th><th colspan="3">Finetune</th> |
|
</tr> |
|
<tr align="center"> |
|
<td>Metric</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td> |
|
</tr> |
|
<tr align="center"> |
|
<td width="120%">Wukong</td><td>51.7</td><td>78.9</td><td>86.3</td><td>77.4</td><td>94.5</td><td>97.0</td><td>76.1</td><td>94.8</td><td>97.5</td><td>92.7</td><td>99.1</td><td>99.6</td> |
|
</tr> |
|
<tr align="center"> |
|
<td width="120%">R2D2</td><td>60.9</td><td>86.8</td><td>92.7</td><td>84.4</td><td>96.7</td><td>98.4</td><td>77.6</td><td>96.7</td><td>98.9</td><td>95.6</td><td>99.8</td><td>100.0</td> |
|
</tr> |
|
<tr align="center"> |
|
<td width="120%">CN-CLIP</td><td>71.2</td><td>91.4</td><td>95.5</td><td>83.8</td><td>96.9</td><td>98.6</td><td>81.6</td><td>97.5</td><td>98.8</td><td>95.3</td><td>99.7</td><td>100.0</td> |
|
</tr> |
|
</table> |
|
<br> |
|
|
|
**COCO-CN Retrieval**: |
|
<table border="1" width="100%"> |
|
<tr align="center"> |
|
<th>Task</th><th colspan="6">Text-to-Image</th><th colspan="6">Image-to-Text</th> |
|
</tr> |
|
<tr align="center"> |
|
<th>Setup</th><th colspan="3">Zero-shot</th><th colspan="3">Finetune</th><th colspan="3">Zero-shot</th><th colspan="3">Finetune</th> |
|
</tr> |
|
<tr align="center"> |
|
<td>Metric</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td> |
|
</tr> |
|
<tr align="center"> |
|
<td width="120%">Wukong</td><td>53.4</td><td>80.2</td><td>90.1</td><td>74.0</td><td>94.4</td><td>98.1</td><td>55.2</td><td>81.0</td><td>90.6</td><td>73.3</td><td>94.0</td><td>98.0</td> |
|
</tr> |
|
<tr align="center"> |
|
<td width="120%">R2D2</td><td>56.4</td><td>85.0</td><td>93.1</td><td>79.1</td><td>96.5</td><td>98.9</td><td>63.3</td><td>89.3</td><td>95.7</td><td>79.3</td><td>97.1</td><td>98.7</td> |
|
</tr> |
|
<tr align="center"> |
|
<td width="120%">CN-CLIP</td><td>69.2</td><td>89.9</td><td>96.1</td><td>81.5</td><td>96.9</td><td>99.1</td><td>63.0</td><td>86.6</td><td>92.9</td><td>83.5</td><td>97.3</td><td>99.2</td> |
|
</tr> |
|
</table> |
|
<br> |
|
|
|
**Zero-shot Image Classification**: |
|
<table border="1" width="100%"> |
|
<tr align="center"> |
|
<th>Task</th><th>CIFAR10</th><th>CIFAR100</th><th>DTD</th><th>EuroSAT</th><th>FER</th><th>FGVC</th><th>KITTI</th><th>MNIST</th><th>PC</th><th>VOC</th> |
|
</tr> |
|
<tr align="center"> |
|
<td width="150%">GIT</td><td>88.5</td><td>61.1</td><td>42.9</td><td>43.4</td><td>41.4</td><td>6.7</td><td>22.1</td><td>68.9</td><td>50.0</td><td>80.2</td> |
|
</tr> |
|
<tr align="center"> |
|
<td width="150%">ALIGN</td><td>94.9</td><td>76.8</td><td>66.1</td><td>52.1</td><td>50.8</td><td>25.0</td><td>41.2</td><td>74.0</td><td>55.2</td><td>83.0</td> |
|
</tr> |
|
<tr align="center"> |
|
<td width="150%">CLIP</td><td>94.9</td><td>77.0</td><td>56.0</td><td>63.0</td><td>48.3</td><td>33.3</td><td>11.5</td><td>79.0</td><td>62.3</td><td>84.0</td> |
|
</tr> |
|
<tr align="center"> |
|
<td width="150%">Wukong</td><td>95.4</td><td>77.1</td><td>40.9</td><td>50.3</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td><td>-</td> |
|
</tr> |
|
<tr align="center"> |
|
<td width="150%">CN-CLIP</td><td>96.0</td><td>79.7</td><td>51.2</td><td>52.0</td><td>55.1</td><td>26.2</td><td>49.9</td><td>79.4</td><td>63.5</td><td>84.9</td> |
|
</tr> |
|
</table> |
|
<br> |
|
|
|
## Citation |
|
If you find Chinese CLIP helpful, feel free to cite our paper. Thanks for your support! |
|
|
|
``` |
|
@article{chinese-clip, |
|
title={Chinese CLIP: Contrastive Vision-Language Pretraining in Chinese}, |
|
author={Yang, An and Pan, Junshu and Lin, Junyang and Men, Rui and Zhang, Yichang and Zhou, Jingren and Zhou, Chang}, |
|
journal={arXiv preprint arXiv:2211.01335}, |
|
year={2022} |
|
} |
|
``` |
|
<br> |
