Instructions to use ModalityDance/OPTD-Qwen3-4B-Late with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ModalityDance/OPTD-Qwen3-4B-Late with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ModalityDance/OPTD-Qwen3-4B-Late")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ModalityDance/OPTD-Qwen3-4B-Late")
model = AutoModelForCausalLM.from_pretrained("ModalityDance/OPTD-Qwen3-4B-Late", device_map="auto")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use ModalityDance/OPTD-Qwen3-4B-Late with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ModalityDance/OPTD-Qwen3-4B-Late"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ModalityDance/OPTD-Qwen3-4B-Late",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ModalityDance/OPTD-Qwen3-4B-Late

SGLang

How to use ModalityDance/OPTD-Qwen3-4B-Late with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ModalityDance/OPTD-Qwen3-4B-Late" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ModalityDance/OPTD-Qwen3-4B-Late",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ModalityDance/OPTD-Qwen3-4B-Late" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ModalityDance/OPTD-Qwen3-4B-Late",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ModalityDance/OPTD-Qwen3-4B-Late with Docker Model Runner:
```
docker model run hf.co/ModalityDance/OPTD-Qwen3-4B-Late
```

OPTD-Qwen3-4B-Late

This is the On-Policy Transition Distillation (OPTD) student checkpoint from the paper On-Policy Transition Distillation.

Setting	Value
Initialization	`Qwen/Qwen3-4B`
Training updates	50
Intermediate blocks (zero-based)	14 / 19 / 24
Domains	math, coding
Weight format	BF16 safetensors
Generation mode	Native non-thinking (enable_thinking=False)

OPTD retains final-policy OPD and aligns teacher/student log-probability increments across the selected blocks and final output on student-generated rollouts. The practical update uses fixed final-policy top-16 candidate identities and detached transition rewards and destination-layer student probabilities.

This repository contains the full inference model, tokenizer, generation configuration, and training metadata. Optimizer state and distributed training shards are not included. checksums.json records the SHA256 of the uploaded checkpoint/tokenizer files.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "ModalityDance/OPTD-Qwen3-4B-Late"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(
    model_id, torch_dtype=torch.bfloat16, device_map="auto"
)
text = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Solve 2x + 5 = 17."}],
    tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tokenizer(text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_new_tokens=1024, do_sample=True,
                        temperature=0.7, top_p=0.95)
print(tokenizer.decode(output[0, inputs.input_ids.shape[-1]:], skip_special_tokens=True))

Evaluation and limitations

The checkpoint is intended for research on specialist-to-student distillation. Paper comparisons require the original checkpoint, prompt mode, benchmark revisions, sample counts, and graders. A short example generation does not reproduce the paper scores. Outputs can be incorrect or follow undesired instructions; generated code should be evaluated in an isolated sandbox.

License and attribution

The underlying student and specialist models retain their respective upstream license terms. Consult the linked base model's license and notices before redistribution or deployment. No new license grant for upstream model weights is implied by this model card.

Downloads last month: -

Safetensors

Model size

4B params

Tensor type

BF16

Model tree for ModalityDance/OPTD-Qwen3-4B-Late

Base model

Qwen/Qwen3-4B-Base

Finetuned

Qwen/Qwen3-4B

Finetuned

(1195)

this model