appmana/deepseek-v4-flash-vision-exp-int4-int8

DeepSeek V4 Flash Vision Exp INT4/INT8 for RTX 30xx

Converted from the public MIT-licensed deepseek-ai/DeepSeek-V4-Flash-Vision-Exp at 6821d6ad3681a4b137b066b76094fa82ebd0a380 with tools/ampere/dsv4_requant_checkpoint.py --vision-format int8-imma from AppMana/forks-vllm-consumer-nvidia-platforms. The checkpoint keeps its own 43 decoder layers, its three native DSpark draft stages and its image routing parameters; nothing is grafted from another model. Stock vLLM cannot load it.

Served revision	65df4c51a2e724340c7963ccddd1f5199a2575e4 (INT8 IMMA vision tower)
BF16 vision reference	branch bf16-vision-reference, bee7f4e9f445e49067b35d88e4ef3ef3ca9f1a56, identical language and draft weights
Text-only sibling	appmana/deepseek-v4-int4-int8, the same quantization applied to DeepSeek-V4-Flash-0731

Quantization

Routed experts are symmetric INT4 with group size 32 and MSE-selected scales; dense, shared-expert and attention linears are channel-wise INT8 (AllSpark W8A16), with the inverse-RoPE wo_a kept in BF16; the sparse-MLA cache and indexer use the packed int8_ds_mla layout. The vision patch projection, attention projections, MLPs and aligner are signed INT8 weights with FP32 scales per 32 input channels and dynamic INT8 activations, executed on integer tensor cores with INT32 accumulation; vision attention runs INT8 QK and PV products under an FP32 online softmax. Normalization, RoPE, residuals, learned image vectors and nonlinearities stay in floating point. The vllm block of config.json names every kernel, including vision_int8.VisionInt8LinearMethod and vision_int8.vision_attention_int8.

Serving

PP=11 across RTX 3090s with layer partition 4,4,4,4,4,4,4,4,5,5,1 (VLLM_PP_LAYER_PARTITION): rank 0 owns the vision tower, aligner and image embeddings, the last rank the output head and the DSpark stages. Shards preserve that ownership and hold at most 4 GiB of tensor data each.

VLLM_PP_LAYER_PARTITION=4,4,4,4,4,4,4,4,5,5,1 vllm serve appmana/deepseek-v4-flash-vision-exp-int4-int8 \
  --tokenizer-mode deepseek_v4 --reasoning-parser deepseek_v4 \
  --pipeline-parallel-size 11 --distributed-executor-backend ray \
  --max-model-len 1000000 --enable-chunked-prefill --max-num-batched-tokens 1024 \
  --max-num-seqs 8 --limit-mm-per-prompt '{"image":8}' --async-scheduling \
  --speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'

The checkpoint's trained draft block size is unchanged; the serving option selects seven probabilistic speculative tokens.

Benchmarks

Vision, 11 RTX 3090s, 2026-09-14

Eleven nodes, each a Ryzen 9 7950X with one RTX 3090 power limited to 250 W, linked by Thunderbolt, serving as above. A fixed set of 100 images (30 five-character OCR codes, 30 coloured-square positions, 20 bar charts, 20 photographs), each asked for an exact answer, four requests at a time:

Vision tower	Correct	Median request s	p95 request s
INT8 IMMA (65df4c51)	100/100	1.50	3.59
BF16 reference (bee7f4e9)	99/100	1.01	1.89

The IMMA latency in that run was the runtime recompiling its vision kernels for every new image size; the fork no longer specialises those kernels on the token count and compiles them at startup, and this table is refreshed with the next chain run. Also in that run: eight concurrent requests with eight distinct images each, all answered correctly; a stretched-image (3072x384) variant completed with the same four counting errors on both towers; a 990,160-token image prompt answered correctly and reused 989,952 tokens on repeat through the KV connector; a different image at identical geometry and cache salt reused none; every rank kept at least 1.16 GiB of GPU memory free throughout; production PTX shows signed INT8 tensor-core MMA in both vision roles.

Text

Text-only rows for this checkpoint use the same table as the sibling card (tools/ampere/dsv4_needle_matrix_report.py --markdown) and are recorded after each fork merge; the language path is the same code and quantization as appmana/deepseek-v4-int4-int8, whose tables are the current reference.

Validation

Before every rollout the fork compares this checkpoint against the BF16 vision reference on the fixture set above, checks pipeline image and text routing and draft decoding, and proves cache isolation for different images with identical text. Full image identities and placeholder positions take part in the external cache key without changing router token ids.