README_CN.md

中文说明 | English

项目介绍

本项目旨在提供更好的中文CLIP模型。该项目使用的训练数据均为公开可访问的图像URL及相关中文文本描述，总量达到400M。经过筛选后，我们最终使用了100M的数据进行训练。 本项目于QQ-ARC Joint Lab, Tencent PCG完成。 更详细的信息可以参考QA-CLIP项目的主页面。我们也在github上开源了模型，QA-CLIP，welcome to star！

实验结果

针对图文检索任务，我们在MUGE Retrieval、Flickr30K-CN和COCO-CN上进行了zero-shot测试。 针对图像零样本分类任务，我们在ImageNet数据集上进行了测试。测试结果见下表：

Flickr30K-CN Zero-shot Retrieval (Official Test Set):

Task	Text-to-Image			Image-to-Text
Metric	R@1	R@5	R@10	R@1	R@5	R@10
CN-CLIPRN50	48.8	76.0	84.6	60.0	85.9	92.0
QA-CLIPRN50	50.5	77.4	86.1	67.1	87.9	93.2
CN-CLIPViT-B/16	62.7	86.9	92.8	74.6	93.5	97.1
QA-CLIPViT-B/16	63.8	88.0	93.2	78.4	96.1	98.5
CN-CLIPViT-L/14	68.0	89.7	94.4	80.2	96.6	98.2
AltClipViT-L/14	69.7	90.1	94.8	84.8	97.7	99.1
QA-CLIPViT-L/14	69.3	90.3	94.7	85.3	97.9	99.2

MUGE Zero-shot Retrieval (Official Validation Set):

Task	Text-to-Image			Image-to-Text
Metric	R@1	R@5	R@10	R@1	R@5	R@10
CN-CLIPRN50	42.6	68.5	78.0	30.0	56.2	66.9
QA-CLIPRN50	44.0	69.9	79.5	32.4	59.5	70.3
CN-CLIPViT-B/16	52.1	76.7	84.4	38.7	65.6	75.1
QA-CLIPViT-B/16	53.2	77.7	85.1	40.7	68.2	77.2
CN-CLIPViT-L/14	56.4	79.8	86.2	42.6	69.8	78.6
AltClipViT-L/14	29.6	49.9	58.8	21.4	42.0	51.9
QA-CLIPViT-L/14	57.4	81.0	87.7	45.5	73.0	81.4

COCO-CN Zero-shot Retrieval (Official Test Set):

Task	Text-to-Image			Image-to-Text
Metric	R@1	R@5	R@10	R@1	R@5	R@10
CN-CLIPRN50	48.1	81.3	90.5	50.9	81.1	90.5
QA-CLIPRN50	50.1	82.5	91.7	56.7	85.2	92.9
CN-CLIPViT-B/16	62.2	87.1	94.9	56.3	84.0	93.3
QA-CLIPViT-B/16	62.9	87.7	94.7	61.5	87.6	94.8
CN-CLIPViT-L/14	64.9	88.8	94.2	60.6	84.4	93.1
AltClipViT-L/14	63.5	87.6	93.5	62.6	88.5	95.9
QA-CLIPViT-L/14	65.7	90.2	95.0	64.5	88.3	95.1

Zero-shot Image Classification on ImageNet:

Task	ImageNet
CN-CLIPRN50	33.5
QA-CLIPRN50	35.5
CN-CLIPViT-B/16	48.4
QA-CLIPViT-B/16	49.7
CN-CLIPViT-L/14	54.7
QA-CLIPViT-L/14	55.8

使用教程

推理代码

推理代码示例：

from PIL import Image
import requests
from transformers import ChineseCLIPProcessor, ChineseCLIPModel

model = ChineseCLIPModel.from_pretrained("TencentARC/QA-CLIP-ViT-B-16")
processor = ChineseCLIPProcessor.from_pretrained("TencentARC/QA-CLIP-ViT-B-16")

url = "https://clip-cn-beijing.oss-cn-beijing.aliyuncs.com/pokemon.jpeg"
image = Image.open(requests.get(url, stream=True).raw)
# Squirtle, Bulbasaur, Charmander, Pikachu in English
texts = ["杰尼龟", "妙蛙种子", "小火龙", "皮卡丘"]

# compute image feature
inputs = processor(images=image, return_tensors="pt")
image_features = model.get_image_features(**inputs)
image_features = image_features / image_features.norm(p=2, dim=-1, keepdim=True)  # normalize

# compute text features
inputs = processor(text=texts, padding=True, return_tensors="pt")
text_features = model.get_text_features(**inputs)
text_features = text_features / text_features.norm(p=2, dim=-1, keepdim=True)  # normalize

# compute image-text similarity scores
inputs = processor(text=texts, images=image, return_tensors="pt", padding=True)
outputs = model(**inputs)
logits_per_image = outputs.logits_per_image  # this is the image-text similarity score
probs = logits_per_image.softmax(dim=1)

致谢

项目代码基于Chinese-CLIP实现，非常感谢他们优秀的开源工作。