JustinLin610
commited on
Commit
•
549ca72
1
Parent(s):
386ceb9
Update README.md
Browse files
README.md
CHANGED
@@ -1,3 +1,117 @@
|
|
1 |
---
|
2 |
license: apache-2.0
|
3 |
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1 |
---
|
2 |
license: apache-2.0
|
3 |
---
|
4 |
+
|
5 |
+
# Chinese-CLIP-Base
|
6 |
+
|
7 |
+
## Introduction
|
8 |
+
This is the base-version of the Chinese CLIP. Chinese CLIP is a simple implementation of CLIP on a large-scale dataset of around 200 million Chinese image-text pairs. For more details, please refer to our technical report https://arxiv.org/abs/2211.01335 and our official github repo https://github.com/OFA-Sys/Chinese-CLIP
|
9 |
+
|
10 |
+
## How to use
|
11 |
+
We provide a simple code snippet to show how to use the API for Chinese-CLIP. For starters, please install cn_clip:
|
12 |
+
```bash
|
13 |
+
# to install the latest stable release
|
14 |
+
pip install cn_clip
|
15 |
+
|
16 |
+
# or install from source code
|
17 |
+
cd Chinese-CLIP/
|
18 |
+
pip install -e .
|
19 |
+
```
|
20 |
+
After installation, use Chinese CLIP as shown below:
|
21 |
+
```python
|
22 |
+
import torch
|
23 |
+
from PIL import Image
|
24 |
+
|
25 |
+
import cn_clip.clip as clip
|
26 |
+
from cn_clip.clip import load_from_name, available_models
|
27 |
+
print("Available models:", available_models())
|
28 |
+
# Available models: ['ViT-B-16', 'ViT-L-14', 'ViT-L-14-336', 'ViT-H-14', 'RN50']
|
29 |
+
|
30 |
+
device = "cuda" if torch.cuda.is_available() else "cpu"
|
31 |
+
model, preprocess = load_from_name("ViT-B-16", device=device, download_root='./')
|
32 |
+
model.eval()
|
33 |
+
image = preprocess(Image.open("examples/pokemon.jpeg")).unsqueeze(0).to(device)
|
34 |
+
text = clip.tokenize(["杰尼龟", "妙蛙种子", "小火龙", "皮卡丘"]).to(device)
|
35 |
+
|
36 |
+
with torch.no_grad():
|
37 |
+
image_features = model.encode_image(image)
|
38 |
+
text_features = model.encode_text(text)
|
39 |
+
# Normalize the features. Please use the normalized features for downstream tasks.
|
40 |
+
image_features /= image_features.norm(dim=-1, keepdim=True)
|
41 |
+
text_features /= text_features.norm(dim=-1, keepdim=True)
|
42 |
+
|
43 |
+
logits_per_image, logits_per_text = model.get_similarity(image, text)
|
44 |
+
probs = logits_per_image.softmax(dim=-1).cpu().numpy()
|
45 |
+
|
46 |
+
print("Label probs:", probs) # [[1.268734e-03 5.436878e-02 6.795761e-04 9.436829e-01]]
|
47 |
+
```
|
48 |
+
|
49 |
+
However, if you are not satisfied with only using the API, feel free to check our github repo https://github.com/OFA-Sys/Chinese-CLIP for more details about training and inference.
|
50 |
+
<br><br>
|
51 |
+
|
52 |
+
## Results
|
53 |
+
### MUGE Text-to-Image Retrieval
|
54 |
+
<table border="1" width="100%">
|
55 |
+
<tr align="center">
|
56 |
+
<th>Setup</th><th colspan="4">Zero-shot</th><th colspan="4">Finetune</th>
|
57 |
+
</tr>
|
58 |
+
<tr align="center">
|
59 |
+
<td>Metric</td><td>R@1</td><td>R@5</td><td>R@10</td><td>MR</td><td>R@1</td><td>R@5</td><td>R@10</td><td>MR</td>
|
60 |
+
</tr>
|
61 |
+
<tr align="center">
|
62 |
+
<td>Wukong<sub>ViT-B</sub></td><td>33.4</td><td>59.3</td><td>69.7</td><td>54.1</td><td>39.2</td><td>66.9</td><td>77.4</td><td>61.2</td>
|
63 |
+
</tr>
|
64 |
+
<tr align="center">
|
65 |
+
<td>R2D2<sub>ViT-B</sub></td><td>-</td><td>-</td><td>-</td><td>-</td><td>47.4</td><td>75.1</td><td>83.5</td><td>68.7</td>
|
66 |
+
</tr>
|
67 |
+
<tr align="center">
|
68 |
+
<td>CN-CLIP<sub>ViT-B</sub></td><td><b>52.1</b></td><td><b>76.7</b></td><td><b>84.4</b></td><td><b>71.1</b></td><td><b>58.4</b></td><td><b>83.6</b></td><td><b>90.0</b></td><td><b>77.4</b></td>
|
69 |
+
</tr>
|
70 |
+
</table>
|
71 |
+
|
72 |
+
|
73 |
+
### Flickr30K-CN Retrieval
|
74 |
+
<table border="1" width="100%">
|
75 |
+
<tr align="center">
|
76 |
+
<th>Task</th><th colspan="6">Text-to-Image</th><th colspan="6">Image-to-Text</th>
|
77 |
+
</tr>
|
78 |
+
<tr align="center">
|
79 |
+
<th>Setup</th><th colspan="3">Zero-shot</th><th colspan="3">Finetune</th><th colspan="3">Zero-shot</th><th colspan="3">Finetune</th>
|
80 |
+
</tr>
|
81 |
+
<tr align="center">
|
82 |
+
<td>Metric</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td>
|
83 |
+
</tr>
|
84 |
+
<tr align="center">
|
85 |
+
<td>Wukong<sub>ViT-B</sub></td><td>45.7</td><td>73.8</td><td>82.2</td><td>67.6</td><td>89.6</td><td>94.2</td><td>66.2</td><td>88.7</td><td>94.3</td><td>83.9</td><td>97.6</td><td>99.0</td>
|
86 |
+
</tr>
|
87 |
+
<tr align="center">
|
88 |
+
<td>R2D2<sub>ViT-B</sub></td><td>-</td><td>-</td><td>-</td><td>78.3</td><td>94.6</td><td>97.0</td><td>-</td><td>-</td><td>-</td><td>92.6</td><td><b>99.1</b></td><td><b>99.8</b></td>
|
89 |
+
</tr>
|
90 |
+
<tr align="center">
|
91 |
+
<td>CN-CLIP<sub>ViT-B</sub></td><td><b>62.7</b></td><td><b>86.9</b></td><td><b>92.8</b></td><td><b>79.1</b></td><td><b>94.8</b></td><td><b>97.4</b></td><td><b>74.6</b></td><td><b>93.5</b></td><td><b>97.1</b></td><td><b>93.5</b></td><td>99.0</td><td>99.5</td>
|
92 |
+
</tr>
|
93 |
+
</table>
|
94 |
+
|
95 |
+
|
96 |
+
### COCO-CN Retrieval
|
97 |
+
<table border="1" width="100%">
|
98 |
+
<tr align="center">
|
99 |
+
<th>Task</th><th colspan="6">Text-to-Image</th><th colspan="6">Image-to-Text</th>
|
100 |
+
</tr>
|
101 |
+
<tr align="center">
|
102 |
+
<th>Setup</th><th colspan="3">Zero-shot</th><th colspan="3">Finetune</th><th colspan="3">Zero-shot</th><th colspan="3">Finetune</th>
|
103 |
+
</tr>
|
104 |
+
<tr align="center">
|
105 |
+
<td>Metric</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td><td>R@1</td><td>R@5</td><td>R@10</td>
|
106 |
+
</tr>
|
107 |
+
<tr align="center">
|
108 |
+
<td>Wukong<sub>ViT-B</sub></td><td>49.2</td><td>79.4</td><td>87.9</td><td>67.0</td><td>91.4</td><td>96.7</td><td>48.3</td><td>77.8</td><td>88.8</td><td>65.8</td><td>90.3</td><td>96.6</td>
|
109 |
+
</tr>
|
110 |
+
<tr align="center">
|
111 |
+
<td>R2D2<sub>ViT-B</sub></td><td>-</td><td>-</td><td>-</td><td>75.1</td><td>94.2</td><td>98.1</td><td>-</td><td>-</td><td>-</td><td>76.1</td><td>95.3</td><td>98.5</td>
|
112 |
+
</tr>
|
113 |
+
<tr align="center">
|
114 |
+
<td>CN-CLIP<sub>ViT-B</sub></td><td><b>62.2</b></td><td><b>86.6</b></td><td><b>94.9</b></td><td><b>77.0</b></td><td><b>97.1</b></td><td><b>99.0</b></td><td><b>57.0</b></td><td><b>84.1</b></td><td><b>93.6</b></td><td><b>77.4</b></td><td><b>96.2</b></td><td><b>98.9</b></td>
|
115 |
+
</tr>
|
116 |
+
</table>
|
117 |
+
<br><br>
|