Instructions to use ATH-MaaS/OvisOCR2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ATH-MaaS/OvisOCR2 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="ATH-MaaS/OvisOCR2")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("ATH-MaaS/OvisOCR2")
model = AutoModelForMultimodalLM.from_pretrained("ATH-MaaS/OvisOCR2", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ATH-MaaS/OvisOCR2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ATH-MaaS/OvisOCR2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ATH-MaaS/OvisOCR2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/ATH-MaaS/OvisOCR2

SGLang

How to use ATH-MaaS/OvisOCR2 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ATH-MaaS/OvisOCR2" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ATH-MaaS/OvisOCR2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ATH-MaaS/OvisOCR2" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ATH-MaaS/OvisOCR2",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use ATH-MaaS/OvisOCR2 with Docker Model Runner:
```
docker model run hf.co/ATH-MaaS/OvisOCR2
```

Inquiry About Multilingual Support and Future Research Plans

by gunoh - opened 7 days ago

Discussion

gunoh

7 days ago

•

edited 7 days ago

Hello,

I am a researcher currently evaluating OvisOCR2 on Korean-language documents.

I have observed cases where the source document clearly contains Korean Hangul text, but the model outputs Chinese characters with the same or a similar meaning instead.

I analyzed the token embeddings and the decoder hidden state at the relevant generation steps. Although the Hangul token showed higher cosine similarity to the decoder hidden state, the Chinese-character token had a larger dot product and was consequently selected as the final output.

I would appreciate your thoughts on the following questions:

Is it possible to mitigate this issue by adjusting the decoding or sampling strategy, without additional fine-tuning?
I understand that the Alibaba Token Hub team has adopted a synthetic data strategy. Are there any plans to invest additional resources in expanding OvisOCR2’s multilingual capabilities in future releases?
Would it be possible to share the source code or pipeline used to generate the synthetic training data? I would be interested in using it to perform post-training and extend the model to additional language domains myself.

Thank you for sharing such a valuable model and for your excellent research.

AyongZheng

3 days ago

Just sharing a personal thought:
cosine-based decoding may mitigate this issue, but it carries a risk of degrading overall performance because the model was trained using dot-product logits.
A less aggressive option would be partial norm correction:

This would still require careful multilingual evaluation, while Korean-focused post-training would likely be more reliable.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment