File size: 3,501 Bytes
357f72e 63aed9f 357f72e aabc31a 357f72e aabc31a 357f72e aabc31a 357f72e 0ed425a 357f72e 5ec5990 357f72e 85e501d 357f72e 85e501d 357f72e d66ce0c 357f72e 63aed9f |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 |
---
license: gemma
language:
- en
pipeline_tag: image-text-to-text
---
# Cerule - A Tiny Mighty Vision Model
### Based on Google's - <span style="color: #D56c76;">Gemma-2b + SigLIP</span>
```
██████╗███████╗██████╗ ██╗ ██╗██╗ ███████╗
██╔════╝██╔════╝██╔══██╗██║ ██║██║ ██╔════╝
██║ █████╗ ██████╔╝██║ ██║██║ █████╗
██║ ██╔══╝ ██╔══██╗██║ ██║██║ ██╔══╝
╚██████╗███████╗██║ ██║╚██████╔╝███████╗███████╗
╚═════╝╚══════╝╚═╝ ╚═╝ ╚═════╝ ╚══════╝╚══════╝
```
We train and release "Cerule", a tiny yet powerful Vision Lanuage Model based on the newly released Google's [Gemma-2b](https://huggingface.co/google/gemma-2b) and Google's [SigLIP](https://huggingface.co/google/siglip-so400m-patch14-384).
We utilise highly efficient data selection techniques with:
```
- Pretraining stage : 650K images (A LAION Subset)
- Finetuning stage : 695K images (SVIT-mix-665K modified for finetuning(Dataset SOON!))
```
The training setup was `4xA100's 80GB` and took ~6 hours to pretrain and ~13 hours to finetune. We modify and adapt the training code from [LLaVA](https://github.com/haotian-liu/LLaVA).
🚨 Training code, Data and more details to release soon!
---
| Image | Example |
|-------|---------|
|  | **Describe the image**<br>The image is a playful and surreal depiction of a man in a space suit, sitting on a chair and holding a green beer bottle. The man is wearing a white space suit, complete with a helmet and gloves. His feet are clad in black and white shoes, and he is placed on a sandy surface. The background features a large, blue planet, with a moon and a star visible in the sky. |
|  | **Who are the characters in the image?**<br>The image features three characters, two of them are Mario and Luigi, and the third one is Yoshi.<br><br>**Describe the actions of the characters**<br>The Mario and Luigi characters are holding their arms out, as if they are waving. Yoshi is standing on its own, with its arms folded. |
|  | **What's funny about this image?**<br>The image is quite humorous as it depicts a man ironing clothes on the back of a yellow taxi cab. This is not a typical sight you'd expect to see in everyday life. |
---
## Loading the model
```
pip install -qr https://huggingface.co/Tensoic/Cerule-v0.1/resolve/main/requirements.txt
```
```python
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Tensoic/Cerule-v0.1", trust_remote_code=True)
```
## Training:
We will release the training code in some time.
### Inference:
Clone the following repo and following instructions for a CLI based inference.
https://github.com/Tensoic-AI/Cerule
## License
Model subject to Gemma(base model license) terms of use along with the underlying datasets(LAOIN and SVIT) subject to their respective licenses. All codes are Apache 2.0
|