Instructions to use XHToken/Spark-X2.5-4B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use XHToken/Spark-X2.5-4B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="XHToken/Spark-X2.5-4B", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("XHToken/Spark-X2.5-4B", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use XHToken/Spark-X2.5-4B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "XHToken/Spark-X2.5-4B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XHToken/Spark-X2.5-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/XHToken/Spark-X2.5-4B

SGLang

How to use XHToken/Spark-X2.5-4B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "XHToken/Spark-X2.5-4B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XHToken/Spark-X2.5-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "XHToken/Spark-X2.5-4B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "XHToken/Spark-X2.5-4B",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use XHToken/Spark-X2.5-4B with Docker Model Runner:
```
docker model run hf.co/XHToken/Spark-X2.5-4B
```

Spark-X2.5 Speed Benchmark

by pengwenzhi - opened 1 day ago

Discussion

pengwenzhi

SparkLLM org 1 day ago

Environment

GPU: NVIDIA H200, 143,771 MiB
NVIDIA driver: 580.105.08
CUDA: 13.0.3

Transformers

PyTorch: 2.13.0+cu130
Transformers: 4.57.6
FlashAttention: 2.8.3
Allocator: PYTORCH_ALLOC_CONF=expandable_segments:True

SGLang

SGLang: 0.0.0.dev1+g20621aa14
PyTorch: 2.13.0+cu130
Transformers: 5.12.1
Container: lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1

Notes

Batch size is 1 and tensor parallel size is 1, each case generates exactly 2,048 tokens.
Context labels from 4K to 1M denote input plus generated tokens; the 1 baseline denotes one input token plus 2,048 generated tokens.
Each case uses one warmup followed by three measured requests on the same server.
Speed is calculated as generated tokens divided by end-to-end generation latency, including both prefill and decoding.
GPU Memory is the maximum of the three request peak values, reported in GiB to match the binary-unit calculation convention commonly labeled GB in benchmark tables. SGLang is sampled with nvidia-smi; Transformers uses torch.cuda.max_memory_allocated().
KV cache dtype is BF16 for all weight quantizations.

1.7B

Model	Input Length	Format	Method / Scheme	GPU Num	Transformers Speed(tokens/s)	Transformers GPU Memory(GB)	SGLang Speed(tokens/s)	SGLang GPU Memory(GB)
Spark-X2.5-1.7B	1	BF16	Baseline	1	57.09	3.26	425.99	5.81
Spark-X2.5-1.7B	1	FP8 W8A8	128×128 per-block	1	-	-	438.72	4.49
Spark-X2.5-1.7B	1	INT8 W8A8	SmoothQuant	1	-	-	439.57	4.51
Spark-X2.5-1.7B	2048	BF16	Baseline	1	57.48	3.45	420.77	6.13
Spark-X2.5-1.7B	2048	FP8 W8A8	128×128 per-block	1	-	-	439.83	4.84
Spark-X2.5-1.7B	2048	INT8 W8A8	SmoothQuant	1	-	-	436.80	4.84
Spark-X2.5-1.7B	6144	BF16	Baseline	1	55.64	3.92	417.47	6.71
Spark-X2.5-1.7B	6144	FP8 W8A8	128×128 per-block	1	-	-	436.11	5.43
Spark-X2.5-1.7B	6144	INT8 W8A8	SmoothQuant	1	-	-	432.05	5.44
Spark-X2.5-1.7B	14336	BF16	Baseline	1	56.97	4.87	405.97	7.16
Spark-X2.5-1.7B	14336	FP8 W8A8	128×128 per-block	1	-	-	423.20	5.88
Spark-X2.5-1.7B	14336	INT8 W8A8	SmoothQuant	1	-	-	421.20	5.89
Spark-X2.5-1.7B	30720	BF16	Baseline	1	53.22	6.77	385.47	8.03
Spark-X2.5-1.7B	30720	FP8 W8A8	128×128 per-block	1	-	-	406.30	6.75
Spark-X2.5-1.7B	30720	INT8 W8A8	SmoothQuant	1	-	-	402.88	6.76
Spark-X2.5-1.7B	129024	BF16	Baseline	1	52.22	18.17	282.52	13.30
Spark-X2.5-1.7B	129024	FP8 W8A8	128×128 per-block	1	-	-	298.91	12.04
Spark-X2.5-1.7B	129024	INT8 W8A8	SmoothQuant	1	-	-	302.38	12.05
Spark-X2.5-1.7B	260096	BF16	Baseline	1	45.08	33.36	182.53	20.32
Spark-X2.5-1.7B	260096	FP8 W8A8	128×128 per-block	1	-	-	194.60	19.04
Spark-X2.5-1.7B	260096	INT8 W8A8	SmoothQuant	1	-	-	196.91	19.05
Spark-X2.5-1.7B	522240	BF16	Baseline	1	28.89	63.74	86.19	34.37
Spark-X2.5-1.7B	522240	FP8 W8A8	128×128 per-block	1	-	-	90.57	33.12
Spark-X2.5-1.7B	522240	INT8 W8A8	SmoothQuant	1	-	-	91.52	33.12
Spark-X2.5-1.7B	1046528	BF16	Baseline	1	11.84	124.51	30.20	63.34
Spark-X2.5-1.7B	1046528	FP8 W8A8	128×128 per-block	1	-	-	31.30	62.06
Spark-X2.5-1.7B	1046528	INT8 W8A8	SmoothQuant	1	-	-	31.53	62.07

4B

Model	Input Length	Format	Method / Scheme	GPU Num	Transformers Speed(tokens/s)	Transformers GPU Memory(GB)	SGLang Speed(tokens/s)	SGLang GPU Memory(GB)	Note
Spark-X2.5-4B	1	BF16	Baseline	1	44.79	7.82	249.76	10.67
Spark-X2.5-4B	1	FP8 W8A8	128×128 per-block	1	-	-	278.55	7.21
Spark-X2.5-4B	1	INT8 W8A8	SmoothQuant	1	-	-	281.36	7.22
Spark-X2.5-4B	2048	BF16	Baseline	1	44.44	8.15	249.46	11.32
Spark-X2.5-4B	2048	FP8 W8A8	128×128 per-block	1	-	-	276.47	7.88
Spark-X2.5-4B	2048	INT8 W8A8	SmoothQuant	1	-	-	279.35	7.89
Spark-X2.5-4B	6144	BF16	Baseline	1	44.24	9.08	242.14	12.46
Spark-X2.5-4B	6144	FP8 W8A8	128×128 per-block	1	-	-	270.72	9.07
Spark-X2.5-4B	6144	INT8 W8A8	SmoothQuant	1	-	-	273.97	9.09
Spark-X2.5-4B	14336	BF16	Baseline	1	45.02	10.92	234.78	13.60
Spark-X2.5-4B	14336	FP8 W8A8	128×128 per-block	1	-	-	263.39	10.21
Spark-X2.5-4B	14336	INT8 W8A8	SmoothQuant	1	-	-	267.28	10.23
Spark-X2.5-4B	30720	BF16	Baseline	1	44.48	14.62	219.81	15.85
Spark-X2.5-4B	30720	FP8 W8A8	128×128 per-block	1	-	-	245.75	12.47
Spark-X2.5-4B	30720	INT8 W8A8	SmoothQuant	1	-	-	248.51	12.48
Spark-X2.5-4B	129024	BF16	Baseline	1	38.70	36.80	144.51	29.37
Spark-X2.5-4B	129024	FP8 W8A8	128×128 per-block	1	-	-	160.09	25.99
Spark-X2.5-4B	129024	INT8 W8A8	SmoothQuant	1	-	-	163.30	26.00
Spark-X2.5-4B	260096	BF16	Baseline	1	27.85	66.36	85.25	47.39
Spark-X2.5-4B	260096	FP8 W8A8	128×128 per-block	1	-	-	92.64	44.01
Spark-X2.5-4B	260096	INT8 W8A8	SmoothQuant	1	-	-	93.84	44.02
Spark-X2.5-4B	522240	BF16	Baseline	1	14.83	125.50	36.78	83.47
Spark-X2.5-4B	522240	FP8 W8A8	128×128 per-block	1	-	-	38.98	80.08
Spark-X2.5-4B	522240	INT8 W8A8	SmoothQuant	1	-	-	39.16	80.10
Spark-X2.5-4B	1046528	BF16	Baseline	1	-	-	-	-	OOM
Spark-X2.5-4B	1046528	FP8 W8A8	128×128 per-block	1	-	-	-	-	OOM
Spark-X2.5-4B	1046528	INT8 W8A8	SmoothQuant	1	-	-	-	-	OOM

[SGLang Setting]: tp_size=1, mem_fraction_static=0.95, context_length=max_total_tokens=input_length+2048, max_running_requests=1, kv_cache_dtype=bfloat16.
[Sampling Config]: max_new_tokens=2048, temperature=0.0, ignore_eos=True.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment