File size: 1,799 Bytes
a22f97a dfd9ec1 a22f97a |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 |
---
license: apache-2.0
datasets:
- UCSC-VLAA/Recap-DataComp-1B
---
# Model Card for ViT-H-14-CLIPS-224-Recap-DataComp-1B
## Model Details
<!-- Provide the basic links for the model. -->
- **Repository:** https://github.com/UCSC-VLAA/CLIPS
- **Paper:** https://arxiv.org/abs/2411.16828
- **Project Page:** https://ucsc-vlaa.github.io/CLIPS/
## Model Usage
### With OpenCLIP
#### Note:
#### 1. We made modifications to the tokenizer implementation in open_clip/tokenizer.py.
#### 2. Due to differences in the default epsilon values for LayerNorm initialization between JAX and PyTorch, we adjusted the default epsilon value in open_clip/transformer.py to align the model's behavior.
#### For more details, refer to https://github.com/UCSC-VLAA/CLIPS.
```
import torch
import torch.nn.functional as F
from urllib.request import urlopen
from PIL import Image
from open_clip import create_model_from_pretrained, get_tokenizer
model, preprocess = create_model_from_pretrained('hf-hub:UCSC-VLAA/ViT-H-14-CLIPS-224-Recap-DataComp-1B')
tokenizer = get_tokenizer('hf-hub:UCSC-VLAA/ViT-H-14-CLIPS-224-Recap-DataComp-1B')
image = Image.open(urlopen(
'https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/beignets-task-guide.png'
))
image = preprocess(image).unsqueeze(0)
text = tokenizer(["a diagram", "a dog", "a cat", "a beignet"], context_length=model.context_length)
with torch.no_grad(), torch.cuda.amp.autocast():
image_features = model.encode_image(image)
text_features = model.encode_text(text)
image_features = F.normalize(image_features, dim=-1)
text_features = F.normalize(text_features, dim=-1)
text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
print("Label probs:", text_probs) # prints: [[0., 0., 0., 1.0]]
``` |