DeepSeek-V4-Flash (EasyDeL)

DeepSeek-V4-Flash converted to the EasyDeL checkpoint format for JAX training and serving.

Weights are bfloat16. Nothing here is pre-quantized — quantization, if you want it, is applied at load time (see below).

Architecture

layers	43
hidden size	4096
attention heads	64, head dim 512, q-LoRA rank 1024
routed experts	256, top-6 per token, 1 shared expert
expert intermediate	2048
vocabulary	129,280 (untied output embedding)
max positions	1,048,576
hyper-connection streams	4

Attention is mixed per layer — sliding, compressed-sparse (rate 4) and heavily-compressed (rate 128) — with a DSA indexer selecting index_topk=512 compressed entries. MLPs are hash-routed MoE.

Loading

import easydel as ed
from jax import numpy as jnp

model = ed.AutoEasyDeLModelForCausalLM.from_pretrained(
    "EasyDeL/DeepSeek-V4-Flash",
    dtype=jnp.bfloat16,
    param_dtype=jnp.bfloat16,
    sharding_axis_dims=(1, 1, 1, 4, 1, 1),   # pp, dp, fsdp, ep, tp, sp
)

sharding_axis_dims follows EasyDeL's axis order (pp, dp, fsdp, ep, tp, sp), where -1 fills the remaining devices. This is a mixture-of-experts model, so expert parallelism (the ep axis) is usually the natural way to split it.

Quantized serving

model = ed.AutoEasyDeLModelForCausalLM.from_pretrained(
    "EasyDeL/DeepSeek-V4-Flash",
    dtype=jnp.bfloat16,
    param_dtype=jnp.bfloat16,
    sharding_axis_dims=(1, 1, 1, 4, 1, 1),
    quantization_config=ed.EasyDeLQuantizationConfig(
        dtype=ed.layers.quantization.QuantizationType.CHANNELWISE, bits=4
    ),
    apply_quantization=True,
)

Stacked-expert linears accept CHANNELWISE. Block formats such as MXFP4 are declined by those layers, which would leave the MoE weights — most of the model — in bfloat16.

Checkpoint layout

EasyDeL/tensorstore format: zarr arrays under model/, not safetensors.

config.json carries fused_param_tp: 1, meaning fused projections are stored in canonical tp=1 order. EasyDeL re-interleaves them on load for whatever tensor-parallel size you run at, so use a recent release.

License

Follows the license of the original DeepSeek-V4 release.