DiffusionGemma 26B E38 — Abliterated BF16
E38 is a deeply tested middle-layer abliteration of Google's DiffusionGemma 26B A4B model.
This BF16 checkpoint is the frozen high-precision E38 reference model used to create the deployment-oriented E38 NVIDIA-style NVFP4 release.
❤️ Help Support the Next Stage of Research
If you like this model, find the testing useful, or want to help determine how much more potential can be extracted from DiffusionGemma, please consider donating.
The current research is moving beyond abliteration and quantization into a much deeper question:
Does DiffusionGemma already contain greater language-quality capability than current diffusion inference is reliably extracting from it?
Current evidence makes this worth investigating, but answering it properly requires a substantial amount of time, compute, experimentation, and manual analysis.
Rare language failures cannot be understood from a handful of prompts.
A promising idea has to go through:
implementation → controlled generation → trajectory analysis → grammar auditing → capability testing → performance testing → fresh validation
A change that improves a few sentences is not enough.
It also has to preserve:
- reasoning
- mathematics
- instruction following
- multimodal capability
- long-form coherence
- output stability
- latency
- throughput
This next stage requires hundreds or thousands of controlled generations for each serious hypothesis, along with detailed analysis of the diffusion process itself.
Support helps fund:
- GPU compute
- large prospective validation runs
- long-form grammar testing
- token-level diffusion trajectory capture
- entropy and confidence analysis
- manual and blinded error review
- capability-retention testing
- benchmark reruns
- model hosting
- continued public releases
If this model is useful to you, even a small contribution helps fund the compute and time required to investigate these questions properly.
The goal is to keep the work open: publish the models, report failures as well as successes, and only claim improvements after they survive controlled testing.
📊 At a Glance
| Metric | Base BF16 | E38 BF16 | E38 NVFP4 |
|---|---|---|---|
| Role | Original reference | Validated E38 reference | Deployment release |
| Precision | BF16 | BF16 | NVFP4 / mixed precision |
| Checkpoint size | ~51.68 GB | 51.68 GB | 18.86 GB |
| Abliteration search | — | 80+ candidates tested | Derived from frozen E38 |
| Sealed Base/E38 evaluation | Included in 3,300 generations | Included in 3,300 generations | Not rerun |
| Refusal suite | 651 generations | 651 generations | Not rerun |
| Unique refusal prompts | 217 | 217 | Not rerun |
| Target-refusal generations | 383/402 — 95.27% | 0/402 — 0.00% | Expected similar / not rerun |
| Prompt-majority refusal | 127/134 — 94.78% | 0/134 — 0.00% | Expected similar / not rerun |
| False-refusal generations | 0/249 — 0.00% | 0/249 — 0.00% | Not rerun |
| Frozen matched validation | 128/200 — 64.0% | 134/200 — 67.0% | 137/200 — 68.5% |
| MATH Level 5 test size | 1,324 | 1,324 | Not rerun |
| MATH Level 5 @1280 | 84.06% | 80.51% | Not rerun |
| Long-form study | Included | Included | Included |
| Grammar /10k words | 4.059 | 5.479 | 3.236 |
| Lexical artifacts /10k | 2.243 | 1.865 | 1.387 |
| Multimodal aggregate | 88.67% | 88.67% | Not rerun on sealed set |
| Mean latency | 2.487 s | 2.300 s | 0.962 s |
| Measured throughput | 162.15 tok/s | 166.97 tok/s | 292.59 tok/s |
| Observed NVFP4 GPU utilization | — | — | ~50–60% |
| Modified language tensors | 0 | 20 | Same 20 retained BF16 |
| Vision tensors modified | 0 | 0 | 0 |
Refusal Test Size
The sealed refusal evaluation used:
- 134 unique target-refusal prompts
- 83 unique benign / false-refusal prompts
- 217 unique refusal-related prompts
- 3 generations per prompt
- 402 target generations per model
- 249 benign generations per model
- 651 refusal-related generations per model
- 1,302 total Base + E38 refusal-related generations
The primary E38 result:
383 / 402 Base target refusals → 0 / 402 E38 target refusals
while maintaining:
0 / 249 false refusals for both Base and E38
🔬 Next Research Stage — Unlocking More of DiffusionGemma's Potential
The next major research stage is a deeper investigation into DiffusionGemma's grammar, lexical reliability, and diffusion inference process.
The question has become larger than simply fixing occasional grammar mistakes.
Current Working Theory
DiffusionGemma may have greater language-quality potential than current diffusion inference is consistently extracting from the model.
This is a working hypothesis, not an established conclusion.
The model can already produce strong reasoning and highly coherent long-form responses, but it occasionally finalizes localized:
- grammar errors
- malformed words
- agreement failures
- missing words
- duplicated words
- punctuation failures
- lexical artifacts
The important question is whether these failures are actually limitations of the learned weights — or whether some of them are introduced while converting the model's internal probability distribution into finalized text.
Findings So Far
1. The Original Base Model Shows the Same General Error Class
A prospective long-form study directly compared:
- Base DiffusionGemma BF16
- E38 BF16
- E38 NVFP4
| Model | Grammar Errors /10k Words | Lexical Artifacts /10k |
|---|---|---|
| Base BF16 | 4.059 | 2.243 |
| E38 BF16 | 5.479 | 1.865 |
| E38 NVFP4 | 3.236 | 1.387 |
The Base model therefore demonstrated the same general class of rare grammar and lexical errors before E38 abliteration and before NVFP4 quantization.
The measured E38 BF16 grammar difference versus Base was statistically unresolved:
- Difference: +1.420 errors /10k words
- 95% CI: -0.674 to +3.541
- McNemar p: 0.560
The current evidence therefore does not establish an E38-specific grammar regression.
No confirmed semantic-substitution events occurred in the fresh long-form study.
2. Error Rate Depends Strongly on Position Inside the Diffusion Canvas
One of the strongest findings so far is that language errors were not evenly distributed through generated text.
DiffusionGemma generates using a 256-token diffusion canvas.
The final 25% of each canvas showed approximately:
3.01× the language-error rate of the first 25%.
This general behavior appeared across:
- Base BF16
- E38 BF16
- E38 NVFP4
Importantly, simply reaching later canvases in a long response did not explain the effect.
The stronger signal was position inside each individual canvas.
That distinction matters.
The model weights themselves do not suddenly change at token position 193.
This raises the possibility that part of the remaining language-quality problem occurs during the diffusion refinement and finalization process.
3. Error-Prone Tokens Finish With More Uncertainty
Detailed trajectory analysis compared error positions with matched clean positions.
Error-prone tokens finished with approximately:
- +0.0504 higher final entropy
- -0.0418 lower top-1 / top-2 confidence margin
than matched clean controls.
In simple terms:
The model appeared less certain about many of the tokens that eventually became errors.
Yet those positions were still finalized.
This does not prove that uncertainty causes the error.
It does, however, provide a measurable signal that can be investigated.
4. The Problem Is Not Simply “Not Enough Denoising Steps”
The current validated inference configuration already performs substantial iterative refinement.
Increasing or changing refinement behavior has not yet produced a validated overall solution.
Some interventions improved specific local metrics while introducing costs elsewhere.
That is why the next phase cannot simply optimize one grammar score.
Any proposed solution must also preserve the model's broader capabilities.
5. NVFP4 Did Not Make Language Quality Worse
The NVFP4 derivative reduced the checkpoint from:
51.68 GB → 18.86 GB
while the long-form study measured:
| Model | Grammar /10k | Lexical /10k |
|---|---|---|
| E38 BF16 | 5.479 | 1.865 |
| E38 NVFP4 | 3.236 | 1.387 |
The lower NVFP4 point estimates are encouraging.
However:
They are not sufficient evidence to claim that NVFP4 quantization directly improves grammar.
The important validated conclusion remains that the NVFP4 conversion did not show evidence of language-quality degradation in this study.
Why This Could Matter Beyond Grammar
If the remaining language errors were entirely encoded in the learned model weights, there may be relatively little that inference alone can do.
But if some errors occur because the diffusion process finalizes a sequence before it has reached the best state the model is capable of representing, the opportunity is different.
It could mean that:
some additional capability already exists inside the model and can potentially be recovered through better inference.
That could potentially improve:
- grammar
- lexical reliability
- long-form consistency
- structured generation
- difficult reasoning trajectories
- general output quality
without necessarily:
- increasing parameter count
- retraining the full model
- creating a larger checkpoint
Again, this has not been proven.
The next research stage is designed to test it.
Deep Grammar Research Plan
The next stage will expand long-form evaluation and classify errors at a much finer level.
Target categories include:
- subject / verb agreement
- singular / plural agreement
- determiner errors
- article errors
- missing words
- duplicated words
- malformed clauses
- sentence-boundary problems
- punctuation errors
- quotation errors
- malformed lexical forms
- locally incorrect word choices
- repetition
- cross-sentence coherence failures
Each error can then be connected back to:
- exact token position
- canvas-relative position
- canvas number
- entropy
- top-1 / top-2 margin
- diffusion step
- stability history
- token replacement history
- response length
- generation seed
- model variant
The study will continue comparing:
- Base BF16
- E38 BF16
- E38 NVFP4
so that model-specific effects can be separated from behavior common to DiffusionGemma itself.
Why This Takes So Much Work
Rare failures are much harder to study than common failures.
If an error occurs only a few times per ten thousand words, finding ten cleaner examples proves almost nothing.
A serious experiment requires enough generations for the underlying signal to emerge from normal stochastic variation.
For each promising idea, the workflow may require:
- implementing the change
- generating a development corpus
- collecting token-level traces
- automatically detecting possible errors
- manually reviewing detected cases
- measuring false positives
- comparing against the frozen baseline
- checking reasoning retention
- checking mathematics retention
- checking instruction following
- checking multimodal behavior
- measuring latency and throughput
- creating a fresh holdout
- rerunning the experiment prospectively
- rejecting the idea if it fails to generalize
A result is only useful if it survives that process.
The objective is not to make a demo look better. The objective is to determine whether DiffusionGemma can reproducibly produce better language while retaining the capabilities and performance already established.
❤️ Support This Research
If you like this model, find the testing useful, or want to help explore how far DiffusionGemma can actually be pushed, please consider donating.
Support directly helps fund the compute and time needed for:
- large-scale controlled generation
- deep grammar analysis
- token-level trajectory research
- capability validation
- benchmark reruns
- performance testing
- fresh prospective holdouts
- open model releases
If this work is useful to you, even a small contribution helps make larger and more rigorous experiments possible.
🚀 Recommended Deployment Version
The E38 BF16 checkpoint was intentionally frozen before quantization so that the effects of abliteration and quantization could be evaluated independently.
The validated NVIDIA-style NVFP4 derivative is available here:
Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4
Matched validation found:
- E38 BF16: 134 / 200 — 67.0%
- E38 NVFP4: 137 / 200 — 68.5%
- Difference: +1.5 percentage points
- 95% CI: -2.0 to +5.0 pp
- p = 0.5811
The difference was not statistically significant.
The validated conclusion is that E38 NVFP4 preserved measured E38 BF16 quality while substantially reducing checkpoint size and improving inference performance.
The sealed refusal suite was not directly rerun on NVFP4.
Because the 20 frozen E38-modified tensors remain BF16 and byte-identical in the NVFP4 derivative while quantization primarily affects routed expert weights, refusal behavior is expected to remain broadly similar to E38.
That expectation is not a replacement for a direct sealed refusal rerun.
⚡ NVFP4 Memory and Throughput
The NVFP4 derivative reduced checkpoint size from:
51.68 GB → 18.86 GB
A reduction of approximately:
63.5%
Measured throughput increased from:
166.97 tok/s → 292.59 tok/s
while mean latency decreased from:
2.300 s → 0.962 s
Measured throughput improvement:
~1.75×
The much smaller routed-expert representation likely contributes to improved memory-system efficiency by reducing expert-weight traffic and data movement during inference.
Possible contributors include:
- reduced expert-weight memory traffic
- improved effective memory-bandwidth utilization
- improved cache behavior
- lower data movement per routed expert execution
This is a working performance explanation, not a fully isolated bottleneck analysis.
Performance Headroom May Remain
During the validated NVFP4 workload, the RTX PRO 6000 Blackwell was observed at approximately:
50–60% GPU utilization
Therefore:
292.59 tok/s should be treated as demonstrated current performance, not as the throughput ceiling of E38 NVFP4 or the RTX PRO 6000 Blackwell.
The utilization figure does not imply linear scaling to 100%.
Potential remaining areas for optimization include:
- memory bandwidth utilization
- cache efficiency
- NVFP4 kernel optimization
- FlashInfer execution
- CUDA graph coverage
- scheduling
- batching
- KV-cache behavior
- prompt / prefill processing
- CPU / GPU synchronization
- future vLLM and FlashInfer improvements
No higher throughput figure is claimed until directly measured.
🧪 Testing Summary
E38 was selected and validated through a large controlled research program rather than a small number of hand-picked prompts.
Completed testing includes:
- 80+ controlled abliteration candidate configurations
- 3,300-generation sealed Base vs E38 evaluation
- 1,302 sealed refusal-related Base + E38 generations
- 2,400-generation prospective inference-optimization study
- 1,324-problem matched MATH Level 5 evaluation
- 3,200-generation mathematics seed study
- 432-generation long-form grammar / lexical attribution study
- 200-prompt matched Base / E38 / E38-NVFP4 validation
- full BBH
- full MuSR
- full MMLU-Pro
- full IFEval
- multimodal Base-vs-E38 evaluation
- tensor-integrity auditing
- generation-capacity studies
- dynamic-capacity testing
- seed-sensitivity testing
- NVFP4 quality-retention validation
- NVFP4 latency / throughput validation
- diffusion trajectory analysis
Promising development findings were tested on fresh or sealed holdouts where practical.
Negative results and regressions were retained rather than discarded.
🔵 What Is E38?
E38 is the selected configuration from a controlled Heretic-based abliteration search of:
google/diffusiongemma-26B-A4B-it
More than 80 candidate configurations were evaluated before E38 was frozen.
The final E38 modification affects:
- Layers: 7–16
- Projections:
attn.o_proj+mlp.down_proj - Modified language tensors: 20
- Vision tensors modified: 0
- Precision: BF16
The checkpoint was frozen after selection.
The E38 modification remained unchanged through subsequent NVFP4 conversion and validation.
Frozen E38 Overlay SHA256
9fe83b2fa2a3b7643f6dcc4a8360feb2cd96dcb9f70d215bd2e8f8f9d9d3044b
Frozen Selection Hash
cd69540ec547f2f5cf181d527e8366fe60b49df7cafde38b434c5e42eeeb6175
No E38.1 checkpoint was created.
🔓 Refusal Behavior
The defining E38 behavioral change is its substantial reduction in target refusal behavior.
Sealed Test Design
- 134 unique target-refusal prompts
- 83 unique benign / false-refusal prompts
- 217 unique refusal-related prompts
- 3 generations per prompt
- 402 target generations per model
- 249 benign generations per model
- 651 refusal-related generations per model
- 1,302 Base + E38 refusal-related generations
Results
| Refusal Metric | Base BF16 | E38 BF16 |
|---|---|---|
| Target refusal | 383 / 402 — 95.27% | 0 / 402 — 0.00% |
| Prompt-majority refusal | 127 / 134 — 94.78% | 0 / 134 — 0.00% |
| False refusal | 0 / 249 — 0.00% | 0 / 249 — 0.00% |
The broader sealed evaluation found:
| Capability Metric | Base BF16 | E38 BF16 |
|---|---|---|
| Strict capability | 91.73% | 90.48% |
| Reasoning | 95.63% | 96.83% |
| Multimodal aggregate | 88.67% | 88.67% |
| Multi-turn | 24 / 24 | 24 / 24 |
E38 should therefore not be expected to retain the refusal behavior of the original Google model.
📈 Public Benchmark Results
E38 is not universally stronger than Base.
Its measured capability distribution changes across tasks.
| Benchmark | Base BF16 | E38 BF16 | Delta |
|---|---|---|---|
| IFEval | 67.10% | 64.70% | -2.40 pp |
| BBH | 71.62% | 73.96% | +2.34 pp |
| MuSR | 41.80% | 50.00% | +8.20 pp |
| MMLU-Pro | 49.61% | 51.57% | +1.96 pp |
| MATH Level 5 @1280 | 84.06% | 80.51% | -3.55 pp |
E38 improved measured performance on:
- BBH
- MuSR
- MMLU-Pro
and regressed on:
- IFEval
- MATH Level 5
E38 is therefore best described as a capability redistribution, not a universally stronger checkpoint.
🧮 MATH Level 5
Matched evaluation at 1280-token generation capacity:
- Base BF16: 1113 / 1324 — 84.06%
- E38 BF16: 1066 / 1324 — 80.51%
- Difference: -3.55 percentage points
- 95% CI: -5.59 to -1.44 pp
- McNemar p: 0.00119
Paired outcomes:
- Both correct: 988
- Base-only correct: 125
- E38-only correct: 78
- Both wrong: 133
The mathematics regression is a real measured tradeoff under the tested inference configuration.
Longer generation capacity benefited both models, which is why the final comparison was performed at a matched 1280-token capacity.
🖼️ Multimodal Evaluation
The sealed multimodal comparison found:
| Model | Accuracy |
|---|---|
| Base BF16 | 88.67% |
| E38 BF16 | 88.67% |
No aggregate multimodal capability loss was observed.
The independent E38 tensor audit confirmed:
0 vision tensors modified
⚙️ Recommended Inference Configuration
The retained configuration after a 2,400-generation prospective inference-optimization study is:
t_max = 0.80t_min = 0.4048maximum denoising steps per canvasentropy_bound = 0.1confidence_threshold = 0.005stability_threshold = 1- adaptive stopping enabled
256token diffusion canvas1280token long-form capacity
The schedule resets for each lazy 256-token canvas.
Conventional autoregressive sampling parameters such as temperature, top-p, and top-k do not replace the diffusion schedule.
No tested inference-only alternative has yet established a better overall quality / capability / performance tradeoff.
⚠️ Known Limitations
E38 is experimental and is not universally stronger than Base.
Known considerations include:
- substantially reduced refusal behavior
- weaker exact-format / strict-instruction performance
- lower matched MATH Level 5 performance
- occasional grammar and lexical errors also observed in Base DiffusionGemma
- stochastic diffusion inference
- sensitivity to output capacity and inference configuration
The current long-form evidence did not establish an E38-specific grammar regression.
The possibility that better diffusion inference could unlock additional language quality remains a research hypothesis.
Users should independently evaluate the model for their intended application.
🧬 Model Lineage
Google DiffusionGemma 26B A4B BF16
↓
Heretic / controlled abliteration search
↓
E38 BF16 — frozen validated reference
↓
E38 NVIDIA-style NVFP4 — validated deployment derivative
The BF16 checkpoint is intentionally retained so the effects of abliteration and quantization can be evaluated independently.
🔐 Reproducibility
Upstream Base
google/diffusiongemma-26B-A4B-it
Pinned revision:
f7f5b7f5fa82ffc52addd066915886d497f5517b
E38 Integrity
Overlay SHA256:
9fe83b2fa2a3b7643f6dcc4a8360feb2cd96dcb9f70d215bd2e8f8f9d9d3044b
Selection hash:
cd69540ec547f2f5cf181d527e8366fe60b49df7cafde38b434c5e42eeeb6175
Measured E38 weight difference:
- Relative Frobenius difference:
0.0250308802854 - Maximum absolute weight difference:
0.201904296875
For meaningful comparisons, report:
- exact checkpoint
- checkpoint precision
- GPU
- runtime/backend
- generation capacity
- diffusion configuration
- seed policy
- number of repetitions
- scoring method
Intended Use
This checkpoint is intended for:
- diffusion-LM research
- abliteration research
- capability evaluation
- multimodal experimentation
- quantization reference
- diffusion-inference research
- local model experimentation
- reproducible comparison with the NVFP4 derivative
Safety and Behavior
This model intentionally has substantially reduced refusal behavior.
It may answer requests that the upstream DiffusionGemma checkpoint would refuse.
This release should therefore not be interpreted as retaining the original model's refusal or safety characteristics.
Users should independently evaluate behavior appropriate for their application.
❤️ Support Continued Research
If this model is useful to you and you want to help investigate how much more capability can be extracted from DiffusionGemma, please consider supporting the research.
Support helps fund the compute, validation, analysis, and time needed to run these experiments rigorously and keep the resulting models and findings public.
Upstream Model
Original upstream model:
google/diffusiongemma-26B-A4B-it
Recommended deployment derivative:
Goodoldjam/DiffusionGemma-26B-E38-Abliterated-NVFP4
License
Apache License 2.0.
This repository contains a modified derivative of:
google/diffusiongemma-26B-A4B-it
- Downloads last month
- 12