|
--- |
|
language: |
|
- ru |
|
tags: |
|
- PyTorch |
|
- Text2Image |
|
thumbnail: "https://github.com/sberbank-ai/ru-clip" |
|
--- |
|
|
|
# Model Card: ruCLIP |
|
Disclaimer: The code for using model you can found [here](https://github.com/sberbank-ai/ru-clip). |
|
# Model Details |
|
The ruCLIP model was developed by researchers at SberDevices and Sber AI based on origin OpenAI paper. |
|
# Model Type |
|
The model uses a ViT-B/32 Transformer architecture (initialized from OpenAI checkpoint and freezed while training) as an image encoder and uses [ruGPT3Small](https://github.com/sberbank-ai/ru-gpts) as a text encoder. These encoders are trained to maximize the similarity of (image, text) pairs via a contrastive loss. |
|
# Documents |
|
Our habr [post](https://habr.com/ru/company/sberdevices/blog/564440/). |
|
# Usage |
|
Code for using model you can obtain in our [repo](https://github.com/sberbank-ai/ru-clip). |
|
|
|
``` |
|
from clip.evaluate.utils import ( |
|
get_text_batch, get_image_batch, get_tokenizer, |
|
show_test_images, load_weights_only |
|
) |
|
import torch |
|
|
|
# Load model and tokenizer |
|
model, args = load_weights_only("ViT-B/32-small") |
|
model = model.cuda().float().eval() |
|
tokenizer = get_tokenizer() |
|
# Load test images and prepare for model |
|
images, texts = show_test_images(args) |
|
input_ids, attention_mask = get_text_batch(["Это " + desc for desc in texts], tokenizer, args) |
|
img_input = get_image_batch(images, args.img_transform, args) |
|
# Call model |
|
with torch.no_grad(): |
|
logits_per_image, logits_per_text = model( |
|
img_input={"x": img_input}, |
|
text_input={"x": input_ids, "attention_mask": attention_mask} |
|
) |
|
``` |
|
|
|
# Performance |
|
We evaluate our model on CIFAR100 and CIFAR10 datasets. |
|
|
|
zero-shot classification CIFAR100 top1 accuracy 0.4057; top5 accuracy 0.6975. |
|
|
|
zero-shot classification CIFAR10 top1 accuracy 0.7803; top5 accuracy 0.9834. |
