README.md

metadata

base_model:
  - tokyotech-llm/Llama-3.1-Swallow-8B-v0.1
  - meta-llama/Llama-3.2-11B-Vision-Instruct
  - meta-llama/Llama-3.1-8B
license: llama3.2
language:
  - ja
pipeline_tag: visual-question-answering

Model Information

Llama-3.2-11B-Vision-Instruct-Swallow-8B-Mergeは、Meta/Llama-3.2-11B-Vision-Instructに日本語能力を付加するためにChat Vectorを用いて作成されたKendamarron/Llama-3.2-11B-Vision-Instruct-Swallow-8B-MergeをLoRAでSFTしたモデルです。

Detail

https://zenn.dev/kendama/articles/cd5196a33bc46c

Recipe

Kendamarron/Llama-3.2-11B-Vision-Instruct-Swallow-8B-Merge = Llama-3.2-11B-Vision-Instruct + (Llama-3.1-Swallow-8B-v0.1 - Llama-3.1-8B)

• Vision Model: meta-llama/Llama-3.2-11B-Vision-Instruct
• Base Text Model: meta-llama/Llama-3.1-8B
• Japanese Text Model: tokyotech-llm/Llama-3.1-Swallow-8B-v0.1
• SFT Dataset: Kendamarron/japanese-photo-instruction

License

Llama 3.2 Community License

How to use

import requests
import torch
from PIL import Image
from transformers import MllamaForConditionalGeneration, AutoProcessor

model_id = "Kendamarron/Llama-3.2-11B-Vision-Instruct-Swallow-8B-LoRA"

model = MllamaForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
    device_map="auto",
)
processor = AutoProcessor.from_pretrained(model_id)

url = "https://huggingface.co/datasets/huggingface/documentation-images/resolve/0052a70beed5bf71b92610a43a52df6d286cd5f3/diffusers/rabbit.jpg"
image = Image.open(requests.get(url, stream=True).raw)

messages = [
    {"role": "user", "content": [
        {"type": "image"},
        {"type": "text", "text": "この画像で一句詠んでください。"}
    ]}
]
input_text = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(
    image,
    input_text,
    add_special_tokens=False,
    return_tensors="pt"
).to(model.device)

output = model.generate(**inputs, max_new_tokens=30)
print(processor.decode(output[0]))