Instructions to use ByteDance/Sa2VA-LLaVA-1.5-7B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ByteDance/Sa2VA-LLaVA-1.5-7B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ByteDance/Sa2VA-LLaVA-1.5-7B", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("ByteDance/Sa2VA-LLaVA-1.5-7B", trust_remote_code=True, device_map="auto")

sam2

How to use ByteDance/Sa2VA-LLaVA-1.5-7B with sam2:

# Use SAM2 with images
import torch
from sam2.sam2_image_predictor import SAM2ImagePredictor

predictor = SAM2ImagePredictor.from_pretrained(ByteDance/Sa2VA-LLaVA-1.5-7B)

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    predictor.set_image(<your_image>)
    masks, _, _ = predictor.predict(<input_prompts>)

# Use SAM2 with videos
import torch
from sam2.sam2_video_predictor import SAM2VideoPredictor

predictor = SAM2VideoPredictor.from_pretrained(ByteDance/Sa2VA-LLaVA-1.5-7B)

with torch.inference_mode(), torch.autocast("cuda", dtype=torch.bfloat16):
    state = predictor.init_state(<your_video>)

    # add new prompts and instantly get the output on the same frame
    frame_idx, object_ids, masks = predictor.add_new_points(state, <your_prompts>):

    # propagate the prompts to get masklets throughout the video
    for frame_idx, object_ids, masks in predictor.propagate_in_video(state):
        ...

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ByteDance/Sa2VA-LLaVA-1.5-7B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ByteDance/Sa2VA-LLaVA-1.5-7B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ByteDance/Sa2VA-LLaVA-1.5-7B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ByteDance/Sa2VA-LLaVA-1.5-7B

SGLang

How to use ByteDance/Sa2VA-LLaVA-1.5-7B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ByteDance/Sa2VA-LLaVA-1.5-7B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ByteDance/Sa2VA-LLaVA-1.5-7B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ByteDance/Sa2VA-LLaVA-1.5-7B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ByteDance/Sa2VA-LLaVA-1.5-7B",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ByteDance/Sa2VA-LLaVA-1.5-7B with Docker Model Runner:
```
docker model run hf.co/ByteDance/Sa2VA-LLaVA-1.5-7B
```

Sa2VA-LLaVA-1.5-7B

Sa2VA-LLaVA-1.5-7B is a Sa2VA model built on LLaVA-1.5-7B (CLIP-ViT-L-336 vision encoder + Vicuna-7B language model) with a SAM2 grounding encoder. The MLLM predicts a [SEG] token whose hidden state conditions the SAM2 mask decoder, producing dense image and video referring segmentation alongside open-ended chat. It is intended as a LISA-comparable baseline within the Sa2VA family.

This checkpoint is self-contained: the SAM2 grounding code is vendored into the repository, so it loads with trust_remote_code=True without any extra packages.

Results

Image referring segmentation (cIoU):

RefCOCO val / testA / testB	RefCOCO+ val / testA / testB	RefCOCOg val / test
80.3 / 82.4 / 76.7	73.1 / 78.0 / 66.2	79.2 / 80.1

Video referring segmentation (J&F):

MeViS (val_u)	ReVOS	Ref-DAVIS17
54.8	54.0	74.6

Grounded conversation generation (GCG, val):

AP50	mIoU	Recall
31.2	66.6	43.7

Usage

import torch
from transformers import AutoModel, AutoTokenizer
from PIL import Image

path = "HarborYuan/Sa2VA-LLaVA-1.5-7B"
model = AutoModel.from_pretrained(
    path, torch_dtype=torch.bfloat16, trust_remote_code=True, low_cpu_mem_usage=True,
).eval().cuda()
tokenizer = AutoTokenizer.from_pretrained(path, trust_remote_code=True)

image = Image.open("your_image.jpg").convert("RGB")
out = model.predict_forward(
    image=image,
    text="<image>Please segment the dog in the image.",
    tokenizer=tokenizer,
)
print(out["prediction"])          # text response containing [SEG]
masks = out["prediction_masks"]   # list of boolean masks at the original image size

Load in bfloat16 (the lm_head is kept in higher precision; torch_dtype="auto" mixes dtypes and fails). This model uses a tokenizer (not an AutoProcessor).

For video, pass video=[frame0, frame1, ...] (a list of PIL images) instead of image.

Notes

SAM2 grounding input resolution is 1024.
Built on Sa2VA.

Citation

If you find this project useful in your research, please consider citing:

@article{sa2va,
  title={Sa2VA: Marrying SAM2 with LLaVA for Dense Grounded Understanding of Images and Videos},
  author={Yuan, Haobo and Li, Xiangtai and Zhang, Tao and Sun, Yueyi and Huang, Zilong and Xu, Shilin and Ji, Shunping and Tong, Yunhai and Qi, Lu and Feng, Jiashi and Yang, Ming-Hsuan},
  journal={IEEE TPAMI},
  year={2026}
}

Downloads last month: 20

Safetensors

Model size

7B params

Tensor type

F32