Instructions to use bbin2022/DeepSeek-V4-Flash-BF16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bbin2022/DeepSeek-V4-Flash-BF16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="bbin2022/DeepSeek-V4-Flash-BF16")

# pip install -U transformers accelerate
# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("bbin2022/DeepSeek-V4-Flash-BF16")
model = AutoModelForCausalLM.from_pretrained("bbin2022/DeepSeek-V4-Flash-BF16", device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use bbin2022/DeepSeek-V4-Flash-BF16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bbin2022/DeepSeek-V4-Flash-BF16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bbin2022/DeepSeek-V4-Flash-BF16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/bbin2022/DeepSeek-V4-Flash-BF16

SGLang

How to use bbin2022/DeepSeek-V4-Flash-BF16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "bbin2022/DeepSeek-V4-Flash-BF16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bbin2022/DeepSeek-V4-Flash-BF16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "bbin2022/DeepSeek-V4-Flash-BF16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bbin2022/DeepSeek-V4-Flash-BF16",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use bbin2022/DeepSeek-V4-Flash-BF16 with Docker Model Runner:
```
docker model run hf.co/bbin2022/DeepSeek-V4-Flash-BF16
```

DeepSeek-V4-Flash-BF16

A lossless BF16 dequantization of deepseek-ai/DeepSeek-V4-Flash (revision 60d8d70770c6776ff598c94bb586a859a38244f1) for accelerators without FP8 or FP4 hardware, produced and validated on 16x AMD Instinct MI250X (gfx90a).

What was converted

Original tensors	Format	Converted to
attention, shared experts, output projections	FP8 E4M3 with E8M0 block-128 scales	BF16
256 routed experts per layer	int8-packed FP4 (E2M1) with E8M0 block-32 scales	BF16
norms, router weights, embeddings, head	BF16	unchanged
hyper-connection scalars, attention sinks	FP32	unchanged
`tid2eid` hash routing table	int64	unchanged

The dequantization is exact: FP4 values times power-of-two scales and FP8 E4M3 values times power-of-two scales are all representable in BF16, so this checkpoint reproduces the original weights bit for bit. Sampled tensors were verified against DeepSeek's own inference/convert.py helpers with a maximum absolute difference of 0.0. Tensor names are unchanged; the .scale tensors and the quantization_config / expert_dtype entries of config.json are removed. The MTP head is included.

Size: 542 GiB in 46 safetensors shards, 35,020 tensors.

Serving notes

vLLM 0.31 serves it as DeepseekV4ForCausalLM with --tokenizer-mode deepseek_v4, --kv-cache-dtype fp8 --block-size 256, parsers deepseek_v4.
The model's output projection has 8 LoRA groups, so tensor parallelism is limited to 8. BF16 therefore needs TP8 with 2 pipeline stages on 64 GB devices (about 34 GiB of weights per GPU).
On gfx90a (MI250X/MI210) vLLM 0.31's ROCm decode path has a bug: the untuned-architecture fallback skips the fused inverse RoPE but reports the rows as rotated, so decode output degrades while prefill is correct. A one-line fix in vllm/v1/attention/ops/rocm_aiter_mla_sparse.py (return 0 rotated rows when not on gfx942/gfx950) restores parity with the pure-torch reference (generation logprobs within 0.03 nats). A second patch lets compress-ratio-4 layers use the Triton indexer path on gfx90a for prompts longer than one prefill chunk.
Measured on 16x MI250X with those fixes, single request, 64 output tokens: prefill 3.2k to 5.1k tok/s, decode 6 tok/s eager and 13 tok/s with HIP graphs.

Validation

transformers 5.17 DeepseekV4ForCausalLM on CPU with this checkpoint produces correct greedy output (bf16 autocast is required because the hyper-connection streams are FP32). vLLM CUDA on an H100 and vLLM ROCm on MI250X (with the fix) match that reference on prompt and generation logprobs.

License

Same as the original model (MIT, see LICENSE). Conversion scripts and the vLLM fixes are described in the accompanying write-up.

Downloads last month: 325

Safetensors

Model size

291B params

Tensor type

BF16

F32

I64

Model tree for bbin2022/DeepSeek-V4-Flash-BF16

Base model

deepseek-ai/DeepSeek-V4-Flash

Quantized

(132)

this model