Instructions to use ByteDance/Ouro-1.4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ByteDance/Ouro-1.4B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ByteDance/Ouro-1.4B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("ByteDance/Ouro-1.4B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ByteDance/Ouro-1.4B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ByteDance/Ouro-1.4B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ByteDance/Ouro-1.4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ByteDance/Ouro-1.4B

SGLang

How to use ByteDance/Ouro-1.4B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ByteDance/Ouro-1.4B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ByteDance/Ouro-1.4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ByteDance/Ouro-1.4B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ByteDance/Ouro-1.4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ByteDance/Ouro-1.4B with Docker Model Runner:
```
docker model run hf.co/ByteDance/Ouro-1.4B
```

README: fix stale transformers version guidance (currently recommends a broken version window)

#14

by VemVemRu - opened 3 days ago

base: refs/heads/main

←

from: refs/pr/14

Discussion Files changed

+10

-1

VemVemRu

3 days ago

The README currently recommends transformers==4.54.1 and warns against >=4.56.0, but with the current code revision this guidance is inverted: it points users at the one version window that crashes, and away from the range that works.

Verified empirically (fresh environments, CPU import test of the custom classes, plus full GPU inference on 4.57.1):

transformers version	result
4.52.4 and earlier	❌ `ImportError: cannot import name 'layer_type_validation' from 'transformers.configuration_utils'`, raised by `configuration_ouro.py` (the helper was added to transformers later)
4.54.1 and 4.55.0	❌ `AttributeError: property 'key_cache' of 'UniversalTransformerCache' object has no setter`. `modeling_ouro.py` assigns `self.key_cache = []` in `UniversalTransformerCache.__init__`, but `Cache.key_cache` was a read-only property in this transformers window
4.57.1	✅ config loads, cache initializes, generation runs correctly

This is consistent with the merged ouro-cache-fix credited in the acknowledgments ("resolved a critical compatibility issue with transformers>=4.56.0"). The ⚠️ warning appears to predate that fix and was never updated.

This is also the likely root cause of #12 (AttributeError on the Colab example): following the README's recommendation lands users in the 4.54 to 4.55 window that raises exactly that error. Related: #13 extends compatibility to transformers 5.x. If that merges, the guidance could become "use transformers>=4.56, including 5.x".

Note: Ouro-2.6B and the -Thinking model cards share this same warning text and would benefit from the same correction.

README: fix stale transformers version guidance (currently recommends a broken version window)6049abdf

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

Ready to merge

This branch is ready to get merged automatically.

· Sign up or log in to comment