Instructions to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="WaveCut/YuE2-3B-OrbitQuant-W4A4")

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("WaveCut/YuE2-3B-OrbitQuant-W4A4", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "WaveCut/YuE2-3B-OrbitQuant-W4A4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4

SGLang

How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "WaveCut/YuE2-3B-OrbitQuant-W4A4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "WaveCut/YuE2-3B-OrbitQuant-W4A4",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use WaveCut/YuE2-3B-OrbitQuant-W4A4 with Docker Model Runner:
```
docker model run hf.co/WaveCut/YuE2-3B-OrbitQuant-W4A4
```

YuE2-3B · OrbitQuant W4A4

Music generation from style and lyrics (m-a-p/YuE2-3B) with packed 4-bit transformer weights and activations. Output: 48 kHz stereo.

Model

392 transformer projections in OrbitQuant W4A4 (224 packed modules). Embeddings, norms, auxiliary projections and the audio VAE stay in source precision; output heads are INT8 or W4A4 only in the profiles below.
Checkpoint 3.04 GB vs 7.26 GB BF16 (−58.2%). Separate 0.53 GB VAE unchanged. No fine-tuning or distillation.

Run

hf download WaveCut/YuE2-3B-OrbitQuant-W4A4 --local-dir YuE2-W4A4
cd YuE2-W4A4 && python3.12 -m venv .venv && source .venv/bin/activate
pip install torch==2.10.0 --index-url https://download.pytorch.org/whl/cu128
(cd src && pip install -r requirements.txt)
python src/run.py --profile fast --fuse-rms-quant --kv-cache-dtype int8 --mode fast --output out

`--mode`	Changes	Logits
`off`	reference decode path	reference
`exact`	lookahead decode loop, ABC head cropped to the sampler's classes, model resident during VAE decode, compiled VAE decoder	bit-identical to `off`
`fast`	+ KV handoff to NAR, compiled NAR pointwise ops, INT8 output heads (DP4A GEMV)	KL 0.0015 semantic / 0.0002 ABC
`lowmem`	+ INT8 token embeddings	as `fast`
`turbo`	`lowmem` with W4A4 output heads	KL 0.016 / 0.0009

Requirements: Linux x86_64, Python 3.12, PyTorch 2.10.0 + CUDA 12.8; OrbitQuant 0.9.7 / native 1.0.4 wheels are bundled in src/wheels. Kernels: SM89 (validated on RTX 4090) and SM120 (built, not run). The first request compiles the VAE decoder (~25 s, cached afterwards).

Measurements

RTX 4090, identical input and seed, warm second request in a fresh process.

Profile	Audio	Wall	audio s / wall s	NVML peak	torch peak
`off`	168.9 s	28.27 s	5.97	5.38 GiB	4.63 GiB
`exact`	168.9 s	17.68 s	9.55	5.60 GiB	4.31 GiB
`fast`	140.7 s	14.55 s	9.67	4.93 GiB	3.66 GiB
`lowmem`	152.2 s	15.85 s	9.60	4.60 GiB	3.25 GiB
`turbo`	178.2 s	17.29 s	10.31	4.49 GiB	3.20 GiB

exact matches off output (mean band delta 0.001 dB). INT8/W4A4 heads change sampling at the same seed, so compare those profiles by audio seconds per wall second. Raw data: evaluation/profiles/.

Files

src/: runtime, kernels and kernel sources; src/kernel-binaries.json lists the binaries per architecture, src/runtime-lock.json pins versions, SHA256SUMS covers every file. examples/: prompt file and WAV demos (BF16, W4A4, fast INT8 KV).

License: CC BY-NC 4.0 for the weights (inherited from M-A-P / YuE2); code components keep their own licenses. No commercial rights.

Downloads last month: 998

Safetensors

Model size

2B params

Tensor type

BF16

Model tree for WaveCut/YuE2-3B-OrbitQuant-W4A4

Base model

m-a-p/YuE2-3B

Finetuned

(1)

this model