Instructions to use d0xin/Swift-Qwen3.8-27B-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use d0xin/Swift-Qwen3.8-27B-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="d0xin/Swift-Qwen3.8-27B-FP8")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("d0xin/Swift-Qwen3.8-27B-FP8")
model = AutoModelForMultimodalLM.from_pretrained("d0xin/Swift-Qwen3.8-27B-FP8", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use d0xin/Swift-Qwen3.8-27B-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "d0xin/Swift-Qwen3.8-27B-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "d0xin/Swift-Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/d0xin/Swift-Qwen3.8-27B-FP8

SGLang

How to use d0xin/Swift-Qwen3.8-27B-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "d0xin/Swift-Qwen3.8-27B-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "d0xin/Swift-Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "d0xin/Swift-Qwen3.8-27B-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "d0xin/Swift-Qwen3.8-27B-FP8",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use d0xin/Swift-Qwen3.8-27B-FP8 with Docker Model Runner:
```
docker model run hf.co/d0xin/Swift-Qwen3.8-27B-FP8
```

Swift-Qwen3.8-27B-FP8

FP8 quantization of ukisai/Swift-Qwen3.8-27b.

This is an independent community quantization and is not an official UkisAI release.

The goal of this checkpoint is to preserve the behavior of Swift-Qwen3.8-27B while reducing VRAM requirements and enabling high-throughput inference with SGLang, including speculative decoding with the model's native MTP head or an external DFlash2 draft model.

Model summary

Upstream model: ukisai/Swift-Qwen3.8-27b
Architecture: Qwen3_5ForConditionalGeneration
Quantization format: compressed-tensors
Quantization scheme: FP8_BLOCK
Weight block size: 128 x 128
Activations: dynamic FP8
Activation group size: 128
Quantizer: llmcompressor 0.13.0
Declared context length: 262,144
Checkpoint size: approximately 29 GB
Native MTP components: retained

The checkpoint was produced from the BF16 Swift-Qwen3.8-27B model rather than requantizing an already quantized derivative.

Quantization details

The quantization process used the official Qwen3.8 FP8 configuration as a reference for the block-FP8 layout.

Checkpoint audit:

Item	Count
Total checkpoint tensors	1,199
2D weight tensors	617
FP8 quantization candidates	407
Effectively excluded / preserved modules	626
Incompatible FP8 candidates after validation	0

Matrices that are not compatible with the required 128 x 128 block structure were preserved instead of being forcibly quantized.

The following classes of tensors were intentionally preserved where appropriate:

embeddings
lm_head
normalization parameters
non-2D weights
matrices whose dimensions are incompatible with the FP8 block layout

The native MTP layers are retained. Compatible MTP projection matrices are quantized to FP8, while incompatible components remain unquantized.

Validation

Validated locally on:

NVIDIA RTX PRO 6000 Blackwell 96 GB
SGLang 0.5.19.dev135+ga4ffb996d
compressed-tensors 0.18.0
CUDA-capable Linux deployment
single-GPU tensor parallelism (TP=1)

SGLang successfully loads the checkpoint as:

type=Qwen3_5ForConditionalGeneration
quant=compressed-tensors

Observed target-model weight memory during loading:

28.47 GB

OpenAI-compatible /v1/chat/completions inference was validated successfully.

Multimodal inference has not yet been separately benchmarked for this quantized checkpoint.

Performance

All measurements below are local measurements from a single NVIDIA RTX PRO 6000 Blackwell 96 GB GPU.

They are intended to document this deployment, not to serve as standardized cross-model benchmarks.

Fixed 4,096-token generation

Same prompt and generation setup for all configurations:

Configuration	Median throughput
Swift FP8, target model only	50.45 tok/s
Swift FP8 + native NEXTN/MTP	103.38 tok/s
Swift FP8 + DFlash2, 8 draft tokens	132.62 tok/s

Measured DFlash2 runs:

131.24 tok/s
132.65 tok/s
132.62 tok/s
median: 132.62 tok/s

Compared with target-only generation, DFlash2 produced approximately 2.63x higher output throughput in this test.

Compared with native NEXTN/MTP, DFlash2 was approximately 28% faster.

The DFlash2 draft model is external and is not included in this repository.

Reasoning-heavy agent test

A separate local A/B test used the same diagnostic prompt, sampling parameters, seed, and reasoning setting for both systems.

The prompt asked the model to diagnose an intermittently slow OpenAI-compatible inference deployment with high GPU utilization, large KV cache, speculative decoding, variable context sizes and concurrency-sensitive latency.

Metric	Previous Qwen FP8 production	Swift FP8 + DFlash2
Wall time	193.48 s	139.63 s
Prompt tokens	229	229
Reasoning tokens	14,838	10,716
Completion tokens	22,848	16,170
Finish reason	stop	stop
Effective completion throughput	118.09 tok/s	115.80 tok/s

Observed change:

wall-clock time: approximately -27.8%
reasoning tokens: approximately -27.8%
completion tokens: approximately -29.2%

The main benefit in this test was not higher raw per-token throughput. Swift reached a similarly useful diagnostic answer with substantially fewer reasoning and completion tokens.

This is a local workload test and should not be interpreted as a standardized quality benchmark.

SGLang usage

Basic serving

python -m sglang.launch_server \
  --model-path /path/to/Swift-Qwen3.8-27B-FP8 \
  --served-model-name Swift-Qwen3.8-27B-FP8 \
  --host 0.0.0.0 \
  --port 30000 \
  --attention-backend flashinfer \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder

Native NEXTN / MTP speculative decoding

The retained native MTP head can be used with SGLang:

python -m sglang.launch_server \
  --model-path /path/to/Swift-Qwen3.8-27B-FP8 \
  --served-model-name Swift-Qwen3.8-27B-FP8 \
  --host 0.0.0.0 \
  --port 30000 \
  --attention-backend flashinfer \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

In the validated SGLang build, NEXTN is internally represented through the EAGLE speculative-decoding path.

DFlash2 speculative decoding

Best local throughput was obtained with a compatible external DFlash2 draft checkpoint:

python -m sglang.launch_server \
  --model-path /path/to/Swift-Qwen3.8-27B-FP8 \
  --served-model-name Swift-Qwen3.8-27B-FP8 \
  --host 0.0.0.0 \
  --port 30000 \
  --attention-backend flashinfer \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --speculative-algorithm DFLASH \
  --speculative-draft-model-path /path/to/Qwen3.8-27B-DFlash2 \
  --speculative-num-draft-tokens 8

The DFlash2 weights are not redistributed in this repository.

Because speculative decoding verifies proposed tokens against the target model, the external draft model affects acceptance rate and speed rather than replacing the target model's token distribution.

Notes

This repository contains the quantized target checkpoint only.

It does not include:

a DFlash2 draft checkpoint
the original BF16 Swift checkpoint
SGLang runtime binaries or containers

Performance depends heavily on GPU architecture, SGLang version, attention backend, context length, concurrency, KV-cache configuration and speculative decoding parameters.

License

This checkpoint is derived from:

ukisai/Swift-Qwen3.8-27b

and follows the Swift Open License v1.0 applicable to the upstream model.

Please refer to the upstream repository and its license text for the authoritative licensing terms.

No additional rights to the upstream model are granted by this repository.

Attribution

Original model:

UkisAI
ukisai/Swift-Qwen3.8-27b

FP8 conversion, validation and local performance measurements for this repository were performed independently by the repository maintainer.

Citation

For the underlying Swift model, please cite or reference the upstream project:

@misc{swift-qwen3.8-27b,
  title  = {Swift-Qwen3.8-27B},
  author = {UkisAI},
  year   = {2026},
  url    = {https://huggingface.co/ukisai/Swift-Qwen3.8-27b}
}

Downloads last month: 492

Safetensors

Model size

28B params

Tensor type

BF16

F8_E4M3

Model tree for d0xin/Swift-Qwen3.8-27B-FP8

Base model

Qwen/Qwen3.8-27B

Finetuned

ukisai/Swift-Qwen3.8-27b

Quantized

(9)

this model