Add yaml and adjust table
Browse files
README.md
CHANGED
@@ -1,10 +1,14 @@
|
|
|
|
|
|
|
|
|
|
1 |
## Model card: multilingual-clip
|
2 |
|
3 |
Multilingual-CLIP extends OpenAI's English text encoders to multiple other languages. This model *only* contains the multilingual text encoder. The corresponding image model `ViT-B-32` needs to be retrieved via instructions found on from OpenAI's [CLIP repository on Github](https://github.com/openai/CLIP). We provide a usage example below.
|
4 |
|
5 |
## Requirements
|
6 |
|
7 |
-
To use both the multilingual text encoder and corresponding image encoder, we need to install the packages `multilingual-clip` and `clip
|
8 |
|
9 |
```
|
10 |
pip install multilingual-clip
|
@@ -59,6 +63,7 @@ print("Image features shape:", image_features.shape)
|
|
59 |
## Evaluation results
|
60 |
|
61 |
None of the M-CLIP models have been extensivly evaluated, but testing them on Txt2Img retrieval on the humanly translated MS-COCO dataset, we see the following **R@10** results:
|
|
|
62 |
| Name | En | De | Es | Fr | Zh | It | Pl | Ko | Ru | Tr | Jp |
|
63 |
| ----------------------------------|:-----: |:-----: |:-----: |:-----: | :-----: |:-----: |:-----: |:-----: |:-----: |:-----: |:-----: |
|
64 |
| [OpenAI CLIP Vit-B/32](https://github.com/openai/CLIP)| 90.3 | - | - | - | - | - | - | - | - | - | - |
|
@@ -66,7 +71,7 @@ None of the M-CLIP models have been extensivly evaluated, but testing them on Tx
|
|
66 |
| [LABSE Vit-L/14](https://huggingface.co/M-CLIP/LABSE-Vit-L-14)| 91.6 | 89.6 | 89.5 | 89.9 | 88.9 | 90.1 | 89.8 | 80.8 | 85.5 | 89.8 | 73.9 |
|
67 |
| [XLM-R Large Vit-B/32](https://huggingface.co/M-CLIP/XLM-Roberta-Large-Vit-B-32)| 91.8 | 88.7 | 89.1 | 89.4 | 89.3 | 89.8| 91.4 | 82.1 | 86.1 | 88.8 | 81.0 |
|
68 |
| [XLM-R Vit-L/14](https://huggingface.co/M-CLIP/XLM-Roberta-Large-Vit-L-14)| 92.4 | 90.6 | 91.0 | 90.0 | 89.7 | 91.1 | 91.3 | 85.2 | 85.8 | 90.3 | 81.9 |
|
69 |
-
| [XLM-R Large Vit-B/16+](https://huggingface.co/M-CLIP/XLM-Roberta-Large-Vit-B-16Plus)|
|
70 |
|
71 |
|
72 |
## Training/Model details
|
|
|
1 |
+
---
|
2 |
+
language: multilingual
|
3 |
+
---
|
4 |
+
|
5 |
## Model card: multilingual-clip
|
6 |
|
7 |
Multilingual-CLIP extends OpenAI's English text encoders to multiple other languages. This model *only* contains the multilingual text encoder. The corresponding image model `ViT-B-32` needs to be retrieved via instructions found on from OpenAI's [CLIP repository on Github](https://github.com/openai/CLIP). We provide a usage example below.
|
8 |
|
9 |
## Requirements
|
10 |
|
11 |
+
To use both the multilingual text encoder and corresponding image encoder, we need to install the packages [`multilingual-clip`](https://github.com/FreddeFrallan/Multilingual-CLIP) and [`clip`](https://github.com/openai/CLIP).
|
12 |
|
13 |
```
|
14 |
pip install multilingual-clip
|
|
|
63 |
## Evaluation results
|
64 |
|
65 |
None of the M-CLIP models have been extensivly evaluated, but testing them on Txt2Img retrieval on the humanly translated MS-COCO dataset, we see the following **R@10** results:
|
66 |
+
|
67 |
| Name | En | De | Es | Fr | Zh | It | Pl | Ko | Ru | Tr | Jp |
|
68 |
| ----------------------------------|:-----: |:-----: |:-----: |:-----: | :-----: |:-----: |:-----: |:-----: |:-----: |:-----: |:-----: |
|
69 |
| [OpenAI CLIP Vit-B/32](https://github.com/openai/CLIP)| 90.3 | - | - | - | - | - | - | - | - | - | - |
|
|
|
71 |
| [LABSE Vit-L/14](https://huggingface.co/M-CLIP/LABSE-Vit-L-14)| 91.6 | 89.6 | 89.5 | 89.9 | 88.9 | 90.1 | 89.8 | 80.8 | 85.5 | 89.8 | 73.9 |
|
72 |
| [XLM-R Large Vit-B/32](https://huggingface.co/M-CLIP/XLM-Roberta-Large-Vit-B-32)| 91.8 | 88.7 | 89.1 | 89.4 | 89.3 | 89.8| 91.4 | 82.1 | 86.1 | 88.8 | 81.0 |
|
73 |
| [XLM-R Vit-L/14](https://huggingface.co/M-CLIP/XLM-Roberta-Large-Vit-L-14)| 92.4 | 90.6 | 91.0 | 90.0 | 89.7 | 91.1 | 91.3 | 85.2 | 85.8 | 90.3 | 81.9 |
|
74 |
+
| [XLM-R Large Vit-B/16+](https://huggingface.co/M-CLIP/XLM-Roberta-Large-Vit-B-16Plus)| **95.0** | **93.0** | **93.6** | **93.1** | **94.0** | **93.1** | **94.4** | **89.0** | **90.0** | **93.0** | **84.2** |
|
75 |
|
76 |
|
77 |
## Training/Model details
|