DeepSeek-V4-Flash-0731
looking into it :)
Also downloading to create quantized GGUFs. @bartowski not sure why you didn't create quants below the original MXFP4, but I published scripts that show some examples of how to convert and quantize: https://huggingface.co/tarruda/DeepSeek-V4-Flash-GGUF/tree/main/scripts
Also this new version comes with dspark draft models built in, but we might need to wait for llama.cpp to fully support it https://github.com/ggml-org/llama.cpp/pull/25173#issuecomment-5117724461
https://huggingface.co/bartowski/DeepSeek-V4-Flash-0731-GGUF
uploaded the mxfp4 version which is full quality
I may look into smaller versions, I was under the impression that quantizing the mxfp4 layers resulted in very aggressive degradation, but I may have been wrong, so will investigate for this model
Thanks!
, I was under the impression that quantizing the mxfp4 layers resulted in very aggressive degradation, but I may have been wrong
Give it a shot at my quants: https://huggingface.co/tarruda/DeepSeek-V4-Flash-GGUF
Even the Q2_K works well, but IQ3_XXS seems to be the sweet spot that works on 128G devices.
I'm undecided, I tried going from the current 4.4 bpw to even just 4.0 bpw and got surprisingly big differences in KLD:
====== Perplexity statistics ======
Mean PPL(Q) : 8.250368 ± 0.111366
Mean PPL(base) : 4.977220 ± 0.058168
Cor(ln(PPL(Q)), ln(PPL(base))): 80.77%
Mean ln(PPL(Q)/PPL(base)) : 0.505386 ± 0.007998
Mean PPL(Q)/PPL(base) : 1.657626 ± 0.013258
Mean PPL(Q)-PPL(base) : 3.273148 ± 0.072952
====== KL divergence statistics ======
Mean KLD: 0.728716 ± 0.005622
Maximum KLD: 18.404858
99.9% KLD: 10.750040
99.0% KLD: 6.275336
95.0% KLD: 2.992667
90.0% KLD: 1.878664
Median KLD: 0.309511
10.0% KLD: 0.000662
5.0% KLD: 0.000078
1.0% KLD: 0.000002
0.1% KLD: -0.000000
Minimum KLD: -0.000004
====== Token probability statistics ======
Mean Δp: -8.865 ± 0.113 %
Maximum Δp: 99.649%
99.9% Δp: 88.941%
99.0% Δp: 51.360%
95.0% Δp: 19.118%
90.0% Δp: 8.059%
75.0% Δp: 0.077%
Median Δp: -0.554%
25.0% Δp: -13.332%
10.0% Δp: -41.793%
5.0% Δp: -64.764%
1.0% Δp: -95.421%
0.1% Δp: -99.829%
Minimum Δp: -100.000%
RMS Δp : 26.492 ± 0.147 %
Same top p: 70.470 ± 0.206 %
like I'm sure it's usable I guess, but it really doesn't feel worth it
@bartowski I have uploaded some quants and even the Q2_K has been working very well here. Could you test PPL/KLD of my IQ3_XXS? To me it feels practically the same as using Deepseek v4 flash official API https://huggingface.co/tarruda/DeepSeek-V4-Flash-0731-GGUF/tree/main/IQ3_XXS
@tarruda hm that performed better than expected:
====== Perplexity statistics ======
Mean PPL(Q) : 5.631696 ± 0.068507
Mean PPL(base) : 4.977220 ± 0.058168
Cor(ln(PPL(Q)), ln(PPL(base))): 92.39%
Mean ln(PPL(Q)/PPL(base)) : 0.123539 ± 0.004676
Mean PPL(Q)/PPL(base) : 1.131494 ± 0.005291
Mean PPL(Q)-PPL(base) : 0.654476 ± 0.026711
====== KL divergence statistics ======
Mean KLD: 0.241077 ± 0.002862
Maximum KLD: 14.848409
99.9% KLD: 7.068842
99.0% KLD: 3.234927
95.0% KLD: 1.140221
90.0% KLD: 0.576226
Median KLD: 0.046611
10.0% KLD: 0.000103
5.0% KLD: 0.000015
1.0% KLD: 0.000000
0.1% KLD: -0.000002
Minimum KLD: -0.000004
====== Token probability statistics ======
Mean Δp: -2.481 ± 0.072 %
Maximum Δp: 99.329%
99.9% Δp: 81.362%
99.0% Δp: 41.644%
95.0% Δp: 13.745%
90.0% Δp: 6.404%
75.0% Δp: 0.470%
Median Δp: -0.017%
25.0% Δp: -2.667%
10.0% Δp: -13.508%
5.0% Δp: -28.190%
1.0% Δp: -75.073%
0.1% Δp: -98.125%
Minimum Δp: -99.978%
RMS Δp : 16.208 ± 0.140 %
Same top p: 84.265 ± 0.164 %
So maybe there's some validity to it, need to keep researching, will explore and consider the options
@bartowski can you share the reference logits (upload temporarily so I can download it) and dataset? I'd like to try playing with more quant formulas to see if it is possible to extract better numbers