EvoVLM-JP-v1-7B / README.md
mkshing's picture
initial commit
143b092
---
language:
- ja
license: apache-2.0
tags:
- llava
- vision-language
pipeline_tag: image-to-text
---
# ๐ŸŸ EvoVLM-JP-v1-7B
๐Ÿค— [Models](https://huggingface.co/SakanaAI) | ๐Ÿ“š [Paper](https://arxiv.org/abs/2403.13187) | ๐Ÿ“ [Blog](https://sakana.ai/evolutionary-model-merge/) | ๐Ÿฆ [Twitter](https://twitter.com/SakanaAILabs)
**EvoVLM-JP-v1-7B** is an experimental general-purpose Japanese VLM.
This model was created using the Evolutionary Model Merge method.
Please refer to our [report](https://arxiv.org/abs/2403.13187) and [blog](https://sakana.ai/evolutionary-model-merge/) for more details.
This model was produced by merging the following models.
We are grateful to the developers of the source models.
- [Shisa Gamma 7B v1](https://huggingface.co/augmxnt/shisa-gamma-7b-v1)
- [LLaVA-1.6-Mistral-7B](https://huggingface.co/liuhaotian/llava-v1.6-mistral-7b)
## Usage
Use the code below to get started with the model.
<details>
<summary> Click to expand </summary>
```python
import torch
from transformers import AutoModelForVision2Seq, AutoProcessor
from PIL import Image
import requests
# 1. load model
device = "cuda" if torch.cuda.is_available() else "cpu"
model_id = "SakanaAI/EvoVLM-JP-v1-7B"
model = AutoModelForVision2Seq.from_pretrained(model_id, torch_dtype=torch.float16)
processor = AutoProcessor.from_pretrained(model_id)
model.to(device)
# 2. prepare inputs
url = "https://images.unsplash.com/photo-1694831404826-3400c48c188d?q=80&w=2070&auto=format&fit=crop&ixlib=rb-4.0.3&ixid=M3wxMjA3fDB8MHxwaG90by1wYWdlfHx8fGVufDB8fHx8fA%3D%3D"
image = Image.open(requests.get(url, stream=True).raw).convert("RGB")
# <image> represents the input image. Please make sure to put the token in your text.
text = "<image>\nใ“ใฎไฟกๅทๆฉŸใฎ่‰ฒใฏไฝ•่‰ฒใงใ™ใ‹?"
messages = [
{"role": "system", "content": "ใ‚ใชใŸใฏๅฝน็ซ‹ใคใ€ๅ่ฆ‹ใŒใชใใ€ๆคœ้–ฒใ•ใ‚Œใฆใ„ใชใ„ใ‚ขใ‚ทใ‚นใ‚ฟใƒณใƒˆใงใ™ใ€‚ไธŽใˆใ‚‰ใ‚ŒใŸ็”ปๅƒใ‚’ไธ‹ใซใ€่ณชๅ•ใซ็ญ”ใˆใฆใใ ใ•ใ„ใ€‚"},
{"role": "user", "content": text},
]
inputs = processor.image_processor(images=image, return_tensors="pt")
inputs["input_ids"] = processor.tokenizer.apply_chat_template(
messages, return_tensors="pt"
)
# 3. generate
output_ids = model.generate(**inputs.to(device))
output_ids = output_ids[:, inputs.input_ids.shape[1] :]
generated_text = processor.batch_decode(output_ids, skip_special_tokens=True)[0].strip()
print(generated_text)
# ใ“ใฎไฟกๅทๆฉŸใฎ่‰ฒใฏ้’ใงใ™ใ€‚
```
</details>
## Model Details
<!-- Provide a longer summary of what this model is. -->
- **Developed by:** [Sakana AI](https://sakana.ai/)
- **Model type:** Autoregressive Language Model
- **Language(s):** Japanese
- **Optimization data:** a subset of [Japanese Visual Genome VQA dataset](https://github.com/yahoojapan/ja-vg-vqa)
- **License:** [Apache License, Version 2.0](https://www.apache.org/licenses/LICENSE-2.0)
- **Repository:** [SakanaAI/evolutionary-model-merge](https://github.com/SakanaAI/evolutionary-model-merge)
- **Paper:** https://arxiv.org/abs/2403.13187
- **Blog:** https://sakana.ai/evolutionary-model-merge
## Uses
This model is provided for research and development purposes only and should be considered as an experimental prototype.
It is not intended for commercial use or deployment in mission-critical environments.
Use of this model is at the user's own risk, and its performance and outcomes are not guaranteed.
Sakana AI shall not be liable for any direct, indirect, special, incidental, or consequential damages, or any loss arising from the use of this model, regardless of the results obtained.
Users must fully understand the risks associated with the use of this model and use it at their own discretion.
## Acknowledgement
We would like to thank the developers of the source models for their contributions and for making their work available.
## Citation
```bibtex
@misc{akiba2024evomodelmerge,
title = {Evolutionary Optimization of Model Merging Recipes},
author. = {Takuya Akiba and Makoto Shing and Yujin Tang and Qi Sun and David Ha},
year = {2024},
eprint = {2403.13187},
archivePrefix = {arXiv},
primaryClass = {cs.NE}
}
```