|
--- |
|
language: |
|
- en |
|
pipeline_tag: image-to-text |
|
inference: false |
|
arxiv: 2304.08485 |
|
license: apache-2.0 |
|
datasets: |
|
- liuhaotian/LLaVA-Pretrain |
|
- liuhaotian/LLaVA-Instruct-150K |
|
--- |
|
# LLaVA Model Card |
|
|
|
 |
|
|
|
Below is the model card of TinyLlava model 1.1b. |
|
|
|
Check out also the Google Colab demo to run Llava on a free-tier Google Colab instance: [](https://colab.research.google.com/drive/1XtdA_UoyNzqiEYVR-iWA-xmit8Y2tKV2#scrollTo=DFVZgElEQk3x) |
|
|
|
|
|
## Model details |
|
|
|
**Model type:** |
|
TinyLLaVA is an open-source chatbot trained by fine-tuning TinyLlama on GPT-generated multimodal instruction-following data. |
|
It is an auto-regressive language model, based on the transformer architecture. |
|
|
|
**Paper or resources for more information:** |
|
https://llava-vl.github.io/ |
|
|
|
## How to use the model |
|
|
|
First, make sure to have `transformers >= 4.35.3`. |
|
The model supports multi-image and multi-prompt generation. Meaning that you can pass multiple images in your prompt. Make sure also to follow the correct prompt template (`USER: xxx\nASSISTANT:`) and add the token `<image>` to the location where you want to query images: |
|
|
|
### Using `pipeline`: |
|
|
|
Below we used [`"YouLiXiya/tinyllava-v1.0-1.1b-hf"`](https://huggingface.co/YouLiXiya/tinyllava-v1.0-1.1b-hf) checkpoint. |
|
|
|
```python |
|
from transformers import pipeline |
|
from PIL import Image |
|
import requests |
|
|
|
model_id = "YouLiXiya/tinyllava-v1.0-1.1b-hf" |
|
pipe = pipeline("image-to-text", model=model_id) |
|
url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/transformers/tasks/ai2d-demo.jpg" |
|
|
|
image = Image.open(requests.get(url, stream=True).raw) |
|
prompt = "USER: <image>\nWhat does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud\nASSISTANT:" |
|
|
|
outputs = pipe(image, prompt=prompt, generate_kwargs={"max_new_tokens": 200}) |
|
print(outputs) |
|
{'generated_text': 'USER: \nWhat does the label 15 represent? (1) lava (2) core (3) tunnel (4) ash cloud\nASSISTANT: The label 15 represents lava, which is the type of rock that is formed from molten magma. '} |
|
``` |
|
|
|
### Using pure `transformers`: |
|
|
|
Below is an example script to run generation in `float16` precision on a GPU device: |
|
|
|
```python |
|
import requests |
|
from PIL import Image |
|
|
|
import torch |
|
from transformers import AutoProcessor, LlavaForConditionalGeneration |
|
|
|
model_id = "YouLiXiya/tinyllava-v1.0-1.1b-hf" |
|
|
|
prompt = "USER: <image>\nWhat are these?\nASSISTANT:" |
|
image_file = "http://images.cocodataset.org/val2017/000000039769.jpg" |
|
|
|
model = LlavaForConditionalGeneration.from_pretrained( |
|
model_id, |
|
torch_dtype=torch.float16, |
|
low_cpu_mem_usage=True, |
|
).to(0) |
|
|
|
processor = AutoProcessor.from_pretrained(model_id) |
|
|
|
raw_image = Image.open(requests.get(image_file, stream=True).raw) |
|
inputs = processor(prompt, raw_image, return_tensors='pt').to(0, torch.float16) |
|
|
|
output = model.generate(**inputs, max_new_tokens=200, do_sample=False) |
|
print(processor.decode(output[0][2:], skip_special_tokens=True)) |
|
``` |
|
|
|
### Model optimization |
|
|
|
#### 4-bit quantization through `bitsandbytes` library |
|
|
|
First make sure to install `bitsandbytes`, `pip install bitsandbytes` and make sure to have access to a CUDA compatible GPU device. Simply change the snippet above with: |
|
|
|
```diff |
|
model = LlavaForConditionalGeneration.from_pretrained( |
|
model_id, |
|
torch_dtype=torch.float16, |
|
low_cpu_mem_usage=True, |
|
+ load_in_4bit=True |
|
) |
|
``` |
|
|
|
#### Use Flash-Attention 2 to further speed-up generation |
|
|
|
First make sure to install `flash-attn`. Refer to the [original repository of Flash Attention](https://github.com/Dao-AILab/flash-attention) regarding that package installation. Simply change the snippet above with: |
|
|
|
```diff |
|
model = LlavaForConditionalGeneration.from_pretrained( |
|
model_id, |
|
torch_dtype=torch.float16, |
|
low_cpu_mem_usage=True, |
|
+ use_flash_attention_2=True |
|
).to(0) |
|
``` |
|
|
|
## License |
|
Llama 2 is licensed under the LLAMA 2 Community License, |
|
Copyright (c) Meta Platforms, Inc. All Rights Reserved. |
