Qwen3.5 GatedDeltaNet: Large logit divergence between full-sequence forward and prefill+decode with cache
Author: JWQZ
Pull request: no
Labels: name: bug

### System Info

# Qwen3.5 GatedDeltaNet: Large logit divergence between full-sequence forward and prefill+decode with cache

## System Info

- `transformers` version: 5.9.0
- Platform: macOS (Apple Silicon, arm64)
- Python version: 3.13
- PyTorch version: 2.12.0 (CPU)
- Tokenizers version: 0.22.2
- Using GPU in script?: No
- `causal-conv1d`: **NOT** installed
- `mamba-ssm`: **NOT** installed

## Who can help?

@ArthurZucker @yzhangcs

## Information

- [x] The official example scripts
- [x] My own modified scripts

## Tasks

- [x] An officially supported task in the `examples` folder
- [x] My own task or dataset (give details below)

## Expected Behavior

When computing the logits for the last token of a sequence, the results should be (nearly) identical regardless of whether the model processes the full sequence at once or uses prefill+decode with cache. For reference, Llama-3.2-1B shows max diff = 3.5e-5 and Mamba-130M shows max diff = 2.2e-4.

## Actual Behavior

Qwen3.5-0.8B produces a **max absolute difference of 0.125** between full-sequence forward and prefill+decode. This is ~3 orders of magnitude larger than other models. The error is identical whether loading in bfloat16 or float32, ruling out precision as the cause.

## Minimal Reproduction

```python
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen3.5-0.8B"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="cpu")
model.eval()

input_ids = tokenizer.encode("The capital of France is", return_tensors="pt")

# Mode A: Full sequence, no cache
with torch.no_grad():
    logits_full = model(input_ids=input_ids, use_cache=False).logits[0, -1, :].float()

# Mode B: Prefill (first N-1 tokens) + decode last token with cache
with torch.no_grad():
    out_prefix = model(input_ids=input_ids[:, :-1], use_cache=True)
    logits_cached = model(
        input_ids=input_ids[:, -1:],
        past_key_values=out_prefix.past_key_values,
        use_cache=True,
    ).logits[0, -1, :].float()

diff = torch.abs(logits_full - logits_cached)
print(f"Max diff: {diff.max().item():.6e}")   # 1.250000e-01
print(f"Mean diff: {diff.mean().item():.6e}")  # 1.944993e-02
```

## Detailed Test Results

Two inference modes are compared:

- **Mode A** — Full-sequence forward, no cache (`use_cache=False`)
- **Mode B** — Prefill first N-1 tokens with cache, then decode last token using the cache

Mode B isolates the issue cleanly: the prefill stage uses the same chunk-based computation as Mode A, and only the final decode step switches to the recurrent path.

### Qwen3.5-0.8B (bfloat16)

```
A vs B:  Max=1.250000e-01  Mean=1.944993e-02  Equal=False

Top-5 largest differences:
Index    Full Forward     Prefill+Decode   Abs Diff       Token
6927     -6.343750        -6.218750        1.250000e-01   'oles'
28100    -1.132812        -1.007812        1.250000e-01   'ois'
124639   -9.125000        -9.250000        1.250000e-01   '版主'
107466   -8.937500        -8.812500        1.250000e-01   '了指'
16697    -8.062500        -7.937500        1.250000e-01   'olen'
```

### Qwen3.5-0.8B (float32)

```
A vs B:  Max=1.250000e-01  Mean=1.944993e-02  Equal=False
```

Loading in float32 produces identical error magnitudes, ruling out bf16 precision as the cause.

### Mamba-130M (float32, comparison)

Mamba also uses conv1d + recurrent state (SSM), a similar architectural pattern to GatedDeltaNet. Neither `causal-conv1d` nor `mamba-ssm` is installed, so the torch fallback path is used (same as Qwen3.5).

```
A vs B:  Max=2.231598e-04  Mean=3.242540e-05  Equal=False
```

### Llama-3.2-1B (float32, baseline)

```
A vs B:  Max=3.528595e-05  Mean=5.242007e-06  Equal=False
```

### Per-layer hidden state divergence (Qwen3.5-0.8B, bfloat16, A vs B)

The error grows through layers, with a large jump at the final layer:

```
Layer    Max Abs Diff     Mean Abs Diff
0        0.000000e+00     0.000000e+00     (embedding, no divergence)
1        1.953125e-03     1.360402e-04
2        3.906250e-03     1.846030e-04
3        1.464844e-03     2.327561e-04
4        1.953125e-03     2.701916e-04
5        1.953125e-03     2.991073e-04
6        3.906250e-03     3.104694e-04
7        2.197266e-03     3.373846e-04
8        2.441406e-03     3.151372e-04
9        2.441406e-03     3.224313e-04
10       2.441406e-03     3.342256e-04
11       2.441406e-03     3.449395e-04
12       3.906250e-03     3.726929e-04
13       3.906250e-03     4.318133e-04
14       3.906250e-03     4.789680e-04
15       7.812500e-03     5.791914e-04
16       7.812500e-03     7.362664e-04
17       1.562500e-02     8.317083e-04
18       1.562500e-02     9.588450e-04
19       1.562500e-02     1.106501e-03
20       1.562500e-02     1.208007e-03
21       1.562500e-02     1.413584e-03
22       1.562500e-02     1.631334e-03
23       3.125000e-02     1.877025e-03
24       2.500000e-01     3.162303e-02     (final layer, large jump)
```

## Summary of Observations

1. **The error magnitude is ~3 orders of magnitude larger than Llama and Mamba.** Qwen3.5 shows max diff = 0.125, while Llama shows 3.5e-5 and Mamba shows 2.2e-4.

2. **Float32 does not help.** The error is identical in bf16 and fp32, indicating this is not a floating-point precision issue.

3. **The error accumulates layer by layer**, starting at layer 1 (1.9e-3) and growing through subsequent layers, with a large jump at the final layer (0.25).

## Full Reproduction Script

<details>
<summary>Click to expand full script (tests all three models with detailed output)</summary>

```python
"""
Reproduce: Qwen3.5 GatedDeltaNet produces significantly different logits
between full-sequence forward and prefill+decode with KV cache.

Environment:
    pip install torch transformers

Usage:
    python reproduce_qwen35_kv_cache.py
"""

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer


def test_kv_cache_consistency(model, tokenizer, model_name, dtype):
    """Compare full-sequence forward vs prefill+decode with KV cache."""
    model.eval()

    prompt = "The capital of France is"
    input_ids = tokenizer.encode(prompt, return_tensors="pt")
    seq_len = input_ids.shape[1]

    print(f"\n{'=' * 70}")
    print(f"Model: {model_name} (dtype={dtype})")
    print(f"{'=' * 70}")
    print(f"Input: \"{prompt}\" (seq_len={seq_len})")

    # Mode A: Full sequence, no cache
    with torch.no_grad():
        out_a = model(input_ids=input_ids, use_cache=False)
        logits_a = out_a.logits[0, -1, :].float()

    # Mode B: Prefill (first N-1 tokens) + Decode (last token with cache)
    with torch.no_grad():
        out_prefix = model(input_ids=input_ids[:, :-1], use_cache=True)
        out_b = model(
            input_ids=input_ids[:, -1:],
            past_key_values=out_prefix.past_key_values,
            use_cache=True,
        )
        logits_b = out_b.logits[0, -1, :].float()

    # Print comparison
    diff = torch.abs(logits_a - logits_b)
    print(f"\n  A vs B:  Max={diff.max().item():.6e}  Mean={diff.mean().item():.6e}  Equal={torch.equal(logits_a, logits_b)}")

    # Top differences
    top5 = torch.topk(diff, 5)
    print(f"\n  Top-5 largest differences:")
    print(f"  {'Index':<8} {'Full Forward':<16} {'Prefill+Decode':<16} {'Abs Diff':<14} {'Token'}")
    for i in range(5):
        idx = top5.indices[i].item()
        print(
            f"  {idx:<8} {logits_a[idx].item():<16.6f} {logits_b[idx].item():<16.6f} "
            f"{top5.values[i].item():<14.6e} '{tokenizer.decode([idx])}'"
        )

    # Per-layer hidden state divergence
    print(f"\n  Per-layer hidden state divergence (last token, A vs B):")
    with torch.no_grad():
        out_full = model(input_ids=input_ids, use_cache=False, output_hidden_states=True)
        out_pfx = model(input_ids=input_ids[:, :-1], use_cache=True, output_hidden_states=True)
        out_dec = model(
            input_ids=input_ids[:, -1:],
            past_key_values=out_pfx.past_key_values,
            use_cache=True,
            output_hidden_states=True,
        )

    num_layers = len(out_full.hidden_states)
    print(f"  {'Layer':<8} {'Max Abs Diff':<16} {'Mean Abs Diff':<16}")
    for layer_idx in range(num_layers):
        h_full = out_full.hidden_states[layer_idx][0, -1, :].float()
        h_dec = out_dec.hidden_states[layer_idx][0, -1, :].float()
        d = torch.abs(h_full - h_dec)
        print(f"  {layer_idx:<8} {d.max().item():<16.6e} {d.mean().item():<16.6e}")


def test_mamba_kv_cache_consistency():
    """Test Mamba-130M (also uses conv1d + recurrent state, like GatedDeltaNet)."""
    model_name = "state-spaces/mamba-130m-hf"
    print(f"\n{'=' * 70}")
    print(f"Model: {model_name} (dtype=float32)")
    print(f"Architecture: MambaMixer (conv1d + SSM recurrent state)")
    print(f"{'=' * 70}")

    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(model_name, dtype=torch.float32, device_map="cpu")
    model.eval()

    prompt = "The capital of France is"
    input_ids = tokenizer.encode(prompt, return_tensors="pt")
    seq_len = input_ids.shape[1]
    print(f"Input: \"{prompt}\" (seq_len={seq_len})")

    # Mode A: Full sequence, no cache
    with torch.no_grad():
        out_a = model(input_ids=input_ids, use_cache=False)
        logits_a = out_a.logits[0, -1, :].float()

    # Mode B: Prefill + Decode (Mamba uses cache_params)
    with torch.no_grad():
        out_prefix = model(input_ids=input_ids[:, :-1], use_cache=True)
        cache = out_prefix.cache_params if hasattr(out_prefix, "cache_params") else out_prefix.past_key_values
        out_b = model(input_ids=input_ids[:, -1:], cache_params=cache, use_cache=True)
        logits_b = out_b.logits[0, -1, :].float()

    diff = torch.abs(logits_a - logits_b)
    print(f"\n  A vs B:  Max={diff.max().item():.6e}  Mean={diff.mean().item():.6e}  Equal={torch.equal(logits_a, logits_b)}")


def main():
    print("Loading models...")

    # --- Qwen3.5-0.8B ---
    model_id = "Qwen/Qwen3.5-0.8B"
    tokenizer = AutoTokenizer.from_pretrained(model_id)

    # Test with bfloat16 (model's native dtype)
    model_bf16 = AutoModelForCausalLM.from_pretrained(
        model_id, dtype=torch.bfloat16, device_map="cpu"
    )
    test_kv_cache_consistency(model_bf16, tokenizer, model_id, "bfloat16")
    del model_bf16

    # Test with float32 (to rule out bf16 precision as the cause)
    model_fp32 = AutoModelForCausalLM.from_pretrained(
        model_id, dtype=torch.float32, device_map="cpu"
    )
    test_kv_cache_consistency(model_fp32, tokenizer, model_id, "float32")
    del model_fp32

    # --- Mamba-130M ---
    print("\n\n" + "#" * 70)
    print("# Comparison: Mamba-130M (conv1d + SSM, similar architecture pattern)")
    print("#" * 70)
    test_mamba_kv_cache_consistency()

    # --- Llama-3.2-1B ---
    print("\n\n" + "#" * 70)
    print("# Baseline: Llama-3.2-1B (standard Transformer)")
    print("#" * 70)
    llama_id = "meta-llama/Llama-3.2-1B"
    tokenizer_llama = AutoTokenizer.from_pretrained(llama_id)
    model_llama = AutoModelForCausalLM.from_pretrained(
        llama_id, dtype=torch.float32, device_map="cpu"
    )
    test_kv_cache_consistency(model_llama, tokenizer_llama, llama_id, "float32")


if __name__ == "__main__":
    main()
```

</details>

## Additional Notes

- The model weights are stored in bfloat16 on disk. Loading in float32 does not change the error magnitude.
- `causal-conv1d` and `mamba-ssm` packages are not installed; the torch fallback paths are used for both Qwen3.5 and Mamba.
- Qwen3.5 uses `Qwen3_5GatedDeltaNet` layers which have two computation paths: `torch_chunk_gated_delta_rule` (used during prefill for sequences > 1 token) and `torch_recurrent_gated_delta_rule` (used during decode for single-token steps).



### Who can help?

@ArthurZucker  @Cyrilvallez I don't know if this phenomenon is normal.

### Information

- [x] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

```python
"""
Reproduce: Qwen3.5 GatedDeltaNet produces significantly different logits
between full-sequence forward and prefill+decode with KV cache.

Environment:
    pip install torch transformers

Usage:
    python reproduce_qwen35_kv_cache.py
"""

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer


def test_kv_cache_consistency(model, tokenizer, model_name, dtype):
    """Compare full-sequence forward vs prefill+decode with KV cache."""
    model.eval()

    prompt = "The capital of France is"
    input_ids = tokenizer.encode(prompt, return_tensors="pt")
    seq_len = input_ids.shape[1]

    print(f"\n{'=' * 70}")
    print(f"Model: {model_name} (dtype={dtype})")
    print(f"{'=' * 70}")
    print(f"Input: \"{prompt}\" (seq_len={seq_len})")

    # Mode A: Full sequence, no cache
    with torch.no_grad():
        out_a = model(input_ids=input_ids, use_cache=False)
        logits_a = out_a.logits[0, -1, :].float()

    # Mode B: Prefill (first N-1 tokens) + Decode (last token with cache)
    with torch.no_grad():
        out_prefix = model(input_ids=input_ids[:, :-1], use_cache=True)
        out_b = model(
            input_ids=input_ids[:, -1:],
            past_key_values=out_prefix.past_key_values,
            use_cache=True,
        )
        logits_b = out_b.logits[0, -1, :].float()

    # Print comparison
    diff = torch.abs(logits_a - logits_b)
    print(f"\n  A vs B:  Max={diff.max().item():.6e}  Mean={diff.mean().item():.6e}  Equal={torch.equal(logits_a, logits_b)}")

    # Top differences
    top5 = torch.topk(diff, 5)
    print(f"\n  Top-5 largest differences:")
    print(f"  {'Index':<8} {'Full Forward':<16} {'Prefill+Decode':<16} {'Abs Diff':<14} {'Token'}")
    for i in range(5):
        idx = top5.indices[i].item()
        print(
            f"  {idx:<8} {logits_a[idx].item():<16.6f} {logits_b[idx].item():<16.6f} "
            f"{top5.values[i].item():<14.6e} '{tokenizer.decode([idx])}'"
        )

    # Per-layer hidden state divergence
    print(f"\n  Per-layer hidden state divergence (last token, A vs B):")
    with torch.no_grad():
        out_full = model(input_ids=input_ids, use_cache=False, output_hidden_states=True)
        out_pfx = model(input_ids=input_ids[:, :-1], use_cache=True, output_hidden_states=True)
        out_dec = model(
            input_ids=input_ids[:, -1:],
            past_key_values=out_pfx.past_key_values,
            use_cache=True,
            output_hidden_states=True,
        )

    num_layers = len(out_full.hidden_states)
    print(f"  {'Layer':<8} {'Max Abs Diff':<16} {'Mean Abs Diff':<16}")
    for layer_idx in range(num_layers):
        h_full = out_full.hidden_states[layer_idx][0, -1, :].float()
        h_dec = out_dec.hidden_states[layer_idx][0, -1, :].float()
        d = torch.abs(h_full - h_dec)
        print(f"  {layer_idx:<8} {d.max().item():<16.6e} {d.mean().item():<16.6e}")


def test_mamba_kv_cache_consistency():
    """Test Mamba-130M (also uses conv1d + recurrent state, like GatedDeltaNet)."""
    model_name = "state-spaces/mamba-130m-hf"
    print(f"\n{'=' * 70}")
    print(f"Model: {model_name} (dtype=float32)")
    print(f"Architecture: MambaMixer (conv1d + SSM recurrent state)")
    print(f"{'=' * 70}")

    tokenizer = AutoTokenizer.from_pretrained(model_name)
    model = AutoModelForCausalLM.from_pretrained(model_name, dtype=torch.float32, device_map="cpu")
    model.eval()

    prompt = "The capital of France is"
    input_ids = tokenizer.encode(prompt, return_tensors="pt")
    seq_len = input_ids.shape[1]
    print(f"Input: \"{prompt}\" (seq_len={seq_len})")

    # Mode A: Full sequence, no cache
    with torch.no_grad():
        out_a = model(input_ids=input_ids, use_cache=False)
        logits_a = out_a.logits[0, -1, :].float()

    # Mode B: Prefill + Decode (Mamba uses cache_params)
    with torch.no_grad():
        out_prefix = model(input_ids=input_ids[:, :-1], use_cache=True)
        cache = out_prefix.cache_params if hasattr(out_prefix, "cache_params") else out_prefix.past_key_values
        out_b = model(input_ids=input_ids[:, -1:], cache_params=cache, use_cache=True)
        logits_b = out_b.logits[0, -1, :].float()

    diff = torch.abs(logits_a - logits_b)
    print(f"\n  A vs B:  Max={diff.max().item():.6e}  Mean={diff.mean().item():.6e}  Equal={torch.equal(logits_a, logits_b)}")


def main():
    print("Loading models...")

    # --- Qwen3.5-0.8B ---
    model_id = "Qwen/Qwen3.5-0.8B"
    tokenizer = AutoTokenizer.from_pretrained(model_id)

    # Test with bfloat16 (model's native dtype)
    model_bf16 = AutoModelForCausalLM.from_pretrained(
        model_id, dtype=torch.bfloat16, device_map="cpu"
    )
    test_kv_cache_consistency(model_bf16, tokenizer, model_id, "bfloat16")
    del model_bf16

    # Test with float32 (to rule out bf16 precision as the cause)
    model_fp32 = AutoModelForCausalLM.from_pretrained(
        model_id, dtype=torch.float32, device_map="cpu"
    )
    test_kv_cache_consistency(model_fp32, tokenizer, model_id, "float32")
    del model_fp32

    # --- Mamba-130M ---
    print("\n\n" + "#" * 70)
    print("# Comparison: Mamba-130M (conv1d + SSM, similar architecture pattern)")
    print("#" * 70)
    test_mamba_kv_cache_consistency()

    # --- Llama-3.2-1B ---
    print("\n\n" + "#" * 70)
    print("# Baseline: Llama-3.2-1B (standard Transformer)")
    print("#" * 70)
    llama_id = "meta-llama/Llama-3.2-1B"
    tokenizer_llama = AutoTokenizer.from_pretrained(llama_id)
    model_llama = AutoModelForCausalLM.from_pretrained(
        llama_id, dtype=torch.float32, device_map="cpu"
    )
    test_kv_cache_consistency(model_llama, tokenizer_llama, llama_id, "float32")


if __name__ == "__main__":
    main()
```
### Expected behavior

Described above

Comments:
This might indicate a precision issue somewhere in the eager path. I'd recommend testing with the fast path packages installed to see if that changes things. If so, you could try figuring out where the divergence is coming from within a single layer. Might be quite a tricky problem to fully debug, though - it may not be possible to get full equivalence between standard torch ops and custom kernels!

@Rocketknight1 Thank you for your response. I am currently unable to install fast path packages in my environment. I will try to test it when I have the opportunity.

qwen 3.5 huh, yeah, its got this thing for its gated delta net, the system architecture itself is different from a lot of stuff, can you try something with unsloth? theres a chance that might work, if not then, well, we gotta look further into this

else if the issue is resolved, then lmk so it can be closed

@Rocketknight1 @ArjunSrivastava1 I have successfully installed the fast path packages. Here are the results:
```plain text
Loading models...
Loading weights: 100%|██████████████████████████████████████████████████████████████████████| 320/320 [00:02<00:00, 145.96it/s]

======================================================================
Model: local_rag/models/Qwen3.5-2B (dtype=bfloat16)
======================================================================
Input: "The capital of France is" (seq_len=5)

  A vs B:  Max=1.250000e-01  Mean=2.019091e-02  Equal=False

  Top-5 largest differences:
  Index    Full Forward     Prefill+Decode   Abs Diff       Token
  7021     8.437500         8.562500         1.250000e-01   ' supported'
  5402     4.562500         4.687500         1.250000e-01   ' themselves'
  95057    -1.078125        -0.964844        1.132812e-01   'jong'
  23984    0.574219         0.683594         1.093750e-01   ' NOI'
  21903    -0.687500        -0.578125        1.093750e-01   'olk'

  Per-layer hidden state divergence (last token, A vs B):
  Layer    Max Abs Diff     Mean Abs Diff   
  0        0.000000e+00     0.000000e+00    
  1        9.765625e-04     1.049815e-04    
  2        7.812500e-03     1.985710e-04    
  3        1.562500e-02     3.692918e-04    
  4        1.562500e-02     4.945786e-04    
  5        2.343750e-02     6.607920e-04    
  6        1.562500e-02     6.875787e-04    
  7        1.562500e-02     7.827021e-04    
  8        1.562500e-02     8.057915e-04    
  9        7.812500e-03     7.972494e-04    
  10       7.812500e-03     8.350797e-04    
  11       1.562500e-02     8.468255e-04    
  12       3.125000e-02     9.641238e-04    
  13       4.687500e-02     1.029897e-03    
  14       4.687500e-02     1.143329e-03    
  15       4.687500e-02     1.331823e-03    
  16       3.125000e-02     1.606248e-03    
  17       3.125000e-02     1.821838e-03    
  18       3.125000e-02     2.087921e-03    
  19       1.562500e-02     2.373613e-03    
  20       1.562500e-02     2.644949e-03    
  21       1.562500e-02     3.286637e-03    
  22       1.953125e-02     3.740489e-03    
  23       6.250000e-02     4.390806e-03    
  24       3.125000e-01     2.545914e-02    
Loading weights: 100%|██████████████████████████████████████████████████████████████████████| 320/320 [00:00<00:00, 321.00it/s]

======================================================================
Model: local_rag/models/Qwen3.5-2B (dtype=float32)
======================================================================
Input: "The capital of France is" (seq_len=5)

  A vs B:  Max=1.250000e-01  Mean=2.019091e-02  Equal=False

  Top-5 largest differences:
  Index    Full Forward     Prefill+Decode   Abs Diff       Token
  7021     8.437500         8.562500         1.250000e-01   ' supported'
  5402     4.562500         4.687500         1.250000e-01   ' themselves'
  95057    -1.078125        -0.964844        1.132812e-01   'jong'
  23984    0.574219         0.683594         1.093750e-01   ' NOI'
  21903    -0.687500        -0.578125        1.093750e-01   'olk'

  Per-layer hidden state divergence (last token, A vs B):
  Layer    Max Abs Diff     Mean Abs Diff   
  0        0.000000e+00     0.000000e+00    
  1        9.765625e-04     1.049815e-04    
  2        7.812500e-03     1.985710e-04    
  3        1.562500e-02     3.692918e-04    
  4        1.562500e-02     4.945786e-04    
  5        2.343750e-02     6.607920e-04    
  6        1.562500e-02     6.875787e-04    
  7        1.562500e-02     7.827021e-04    
  8        1.562500e-02     8.057915e-04    
  9        7.812500e-03     7.972494e-04    
  10       7.812500e-03     8.350797e-04    
  11       1.562500e-02     8.468255e-04    
  12       3.125000e-02     9.641238e-04    
  13       4.687500e-02     1.029897e-03    
  14       4.687500e-02     1.143329e-03    
  15       4.687500e-02     1.331823e-03    
  16       3.125000e-02     1.606248e-03    
  17       3.125000e-02     1.821838e-03    
  18       3.125000e-02     2.087921e-03    
  19       1.562500e-02     2.373613e-03    
  20       1.562500e-02     2.644949e-03    
  21       1.562500e-02     3.286637e-03    
  22       1.953125e-02     3.740489e-03    
  23       6.250000e-02     4.390806e-03    
  24       3.125000e-01     2.545914e-02    


######################################################################
# Comparison: Mamba-130M (conv1d + SSM, similar architecture pattern)
######################################################################

======================================================================
Model: local_rag/models/mamba-130m-hf (dtype=float32)
Architecture: MambaMixer (conv1d + SSM recurrent state)
======================================================================
[transformers] The fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. Falling back to the sequential implementation of Mamba, as use_mambapy is set to False. To install follow https://github.com/state-spaces/mamba/#installation for mamba-ssm and install the kernels library using `pip install kernels` or https://github.com/Dao-AILab/causal-conv1d for causal-conv1d. For the mamba.py backend, follow https://github.com/alxndrTL/mamba.py.
Loading weights: 100%|█████████████████████████████████████████████████████████████████████| 242/242 [00:00<00:00, 1408.03it/s]
Input: "The capital of France is" (seq_len=5)

  A vs B:  Max=1.945496e-04  Mean=2.503319e-05  Equal=False


######################################################################
# Baseline: Llama-3.2-1B (standard Transformer)
######################################################################
Loading weights: 100%|███████████████████████████████████████████████████████████████████████| 146/146 [00:32<00:00,  4.44it/s]

======================================================================
Model: local_rag/models/Llama-3.2-1B (dtype=float32)
======================================================================
Input: "The capital of France is" (seq_len=6)

  A vs B:  Max=1.144409e-05  Mean=1.459756e-06  Equal=False

  Top-5 largest differences:
  Index    Full Forward     Prefill+Decode   Abs Diff       Token
[transformers] Ignoring clean_up_tokenization_spaces=True for BPE tokenizer TokenizersBackend. The clean_up_tokenization post-processing step is designed for WordPiece tokenizers and is destructive for BPE (it strips spaces before punctuation). Set clean_up_tokenization_spaces=False to suppress this warning, or set clean_up_tokenization_spaces_for_bpe_even_though_it_will_corrupt_output=True to force cleanup anyway.
  323      14.886902        14.886890        1.144409e-05   ' and'
  91632    6.786200         6.786190         1.049042e-05   ' flushing'
  19073    15.380948        15.380959        1.049042e-05   ' officially'
  12756    12.555473        12.555483        9.536743e-06   ' totally'
  9822     16.255272        16.255281        9.536743e-06   ' France'

  Per-layer hidden state divergence (last token, A vs B):
  Layer    Max Abs Diff     Mean Abs Diff   
  0        0.000000e+00     0.000000e+00    
  1        8.940697e-08     1.204882e-08    
  2        1.490116e-07     1.978947e-08    
  3        2.980232e-07     2.672368e-08    
  4        1.788139e-07     3.705009e-08    
  5        3.576279e-07     4.293952e-08    
  6        2.980232e-07     4.804656e-08    
  7        2.682209e-07     5.575021e-08    
  8        3.278255e-07     6.167988e-08    
  9        3.576279e-07     5.991888e-08    
  10       4.172325e-07     7.358835e-08    
  11       4.768372e-07     8.196093e-08    
  12       7.152557e-07     1.008470e-07    
  13       9.536743e-07     1.248654e-07    
  14       9.536743e-07     1.596618e-07    
  15       1.132488e-06     2.043766e-07    
  16       6.496906e-06     1.211861e-06   
```
Below is my current environment:
```plain text
Package                                  Version             Editable project location
---------------------------------------- ------------------- ----------------------------------------
accelerate                               1.13.0
aiohappyeyeballs                         2.6.1
aiohttp                                  3.13.5
aiosignal                                1.4.0
aiosqlite                                0.22.1
airportsdata                             20260315
aistudio-sdk                             0.3.8
annotated-doc                            0.0.4
annotated-types                          0.7.0
anthropic                                0.71.0
anyio                                    4.13.0
apache-tvm-ffi                           0.1.9
astor                                    0.8.1
attrs                                    26.1.0
bce-python-sdk                           0.9.71
bcrypt                                   5.0.0
blake3                                   1.0.8
cachetools                               7.1.2
causal_conv1d                            1.6.2.post1         /data/cjw/whls/causal-conv1d-1.6.2.post1
cbor2                                    6.1.1
certifi                                  2026.4.22
cffi                                     2.0.0
chardet                                  7.4.3
charset-normalizer                       3.4.7
click                                    8.4.0
cloudpickle                              3.1.2
colorlog                                 6.10.1
compressed-tensors                       0.15.0.1
crc32c                                   2.8
cryptography                             48.0.0
cuda-bindings                            12.9.7
cuda-pathfinder                          1.5.4
cuda-python                              12.9.7
cuda-tile                                1.3.0
cuda-toolkit                             12.9.1
cupy-cuda12x                             14.0.1
Deprecated                               1.3.1
depyf                                    0.20.0
dill                                     0.4.1
diskcache                                5.6.3
distro                                   1.9.0
dnspython                                2.8.0
docstring_parser                         0.18.0
einops                                   0.8.2
email-validator                          2.3.0
faiss-cpu                                1.13.2
fastapi                                  0.136.1
fastapi-cli                              0.0.24
fastapi-cloud-cli                        0.17.1
fastar                                   0.11.0
fastsafetensors                          0.3.1
filelock                                 3.29.0
fla-core                                 0.5.0
flash-linear-attention                   0.5.0
flashinfer-cubin                         0.6.11.post2
flashinfer-python                        0.6.11.post2
frozendict                               2.4.7
frozenlist                               1.8.0
fsspec                                   2026.4.0
future                                   1.0.0
gguf                                     0.19.0
googleapis-common-protos                 1.75.0
grpcio                                   1.80.0
h11                                      0.16.0
hf-xet                                   1.5.0
httpcore                                 1.0.9
httptools                                0.7.1
httpx                                    0.28.1
httpx-sse                                0.4.3
huggingface_hub                          1.15.0
humming-kernels                          0.1.2
idna                                     3.15
ijson                                    3.5.0
imagesize                                2.0.0
importlib_metadata                       8.0.0
interegular                              0.3.3
Jinja2                                   3.1.6
jiter                                    0.14.0
jmespath                                 1.1.0
joblib                                   1.5.3
jsonpatch                                1.33
jsonpointer                              3.1.1
jsonschema                               4.26.0
jsonschema-specifications                2025.9.1
langchain-core                           1.4.0
langchain-protocol                       0.0.15
langchain-text-splitters                 1.1.2
langsmith                                0.8.5
lark                                     1.2.2
llguidance                               1.7.5
llvmlite                                 0.47.0
lm-format-enforcer                       0.11.3
loguru                                   0.7.3
lxml                                     6.1.0
markdown-it-py                           4.2.0
MarkupSafe                               3.0.3
mcp                                      1.27.1
mdurl                                    0.1.2
mistral_common                           1.11.2
ml_dtypes                                0.5.4
model-hosting-container-standards        0.1.15
modelscope                               1.37.1
mpmath                                   1.3.0
msgpack                                  1.1.2
msgspec                                  0.21.1
multidict                                6.7.1
nest-asyncio                             1.6.0
networkx                                 3.6.1
ninja                                    1.13.0
numba                                    0.65.0
numpy                                    2.2.6
nvidia-cublas                            13.0.0.19
nvidia-cublas-cu12                       12.9.1.4
nvidia-cuda-cccl                         13.3.3.3.1
nvidia-cuda-cccl-cu12                    12.6.77
nvidia-cuda-crt                          13.3.33
nvidia-cuda-cupti                        13.0.48
nvidia-cuda-cupti-cu12                   12.9.79
nvidia-cuda-nvcc                         13.3.33
nvidia-cuda-nvrtc                        13.0.48
nvidia-cuda-nvrtc-cu12                   12.9.86
nvidia-cuda-runtime                      13.0.48
nvidia-cuda-runtime-cu12                 12.9.79
nvidia-cudnn-cu12                        9.17.1.4
nvidia-cudnn-cu13                        9.19.0.56
nvidia-cudnn-frontend                    1.18.0
nvidia-cufft                             12.0.0.15
nvidia-cufft-cu12                        11.4.1.4
nvidia-cufile                            1.15.0.42
nvidia-cufile-cu12                       1.14.1.1
nvidia-curand                            10.4.0.35
nvidia-curand-cu12                       10.3.10.19
nvidia-cusolver                          12.0.3.29
nvidia-cusolver-cu12                     11.7.5.82
nvidia-cusparse                          12.6.2.49
nvidia-cusparse-cu12                     12.5.10.65
nvidia-cusparselt-cu12                   0.7.1
nvidia-cusparselt-cu13                   0.8.0
nvidia-cutlass-dsl                       4.5.2
nvidia-cutlass-dsl-libs-base             4.5.2
nvidia-cutlass-dsl-libs-cu13             4.5.2
nvidia-ml-py                             13.595.45
nvidia-nccl-cu12                         2.28.9
nvidia-nccl-cu13                         2.28.9
nvidia-nvjitlink                         13.0.39
nvidia-nvjitlink-cu12                    12.9.86
nvidia-nvshmem-cu12                      3.4.5
nvidia-nvshmem-cu13                      3.4.5
nvidia-nvtx                              13.0.39
nvidia-nvtx-cu12                         12.9.79
nvidia-nvvm                              13.3.33
openai                                   2.37.0
openai-harmony                           0.0.8
opencv-contrib-python                    4.10.0.84
opencv-python-headless                   4.13.0.92
opentelemetry-api                        1.41.1
opentelemetry-exporter-otlp              1.41.1
opentelemetry-exporter-otlp-proto-common 1.41.1
opentelemetry-exporter-otlp-proto-grpc   1.41.1
opentelemetry-exporter-otlp-proto-http   1.41.1
opentelemetry-proto                      1.41.1
opentelemetry-sdk                        1.41.1
opentelemetry-semantic-conventions       0.62b1
opentelemetry-semantic-conventions-ai    0.4.13
opt-einsum                               3.3.0
orjson                                   3.11.9
outlines                                 0.1.11
outlines_core                            0.2.14
packaging                                26.0
paddleocr                                3.6.0
paddlepaddle                             3.3.1
paddlex                                  3.6.1
pandas                                   3.0.3
partial-json-parser                      0.2.1.1.post7
pillow                                   12.2.0
pip                                      26.0.1
prettytable                              3.17.0
prometheus_client                        0.25.0
prometheus-fastapi-instrumentator        7.1.0
propcache                                0.5.2
protobuf                                 6.33.6
psutil                                   7.2.2
py-cpuinfo                               9.0.0
pybase64                                 1.4.3
pyclipper                                1.4.0
pycountry                                26.2.16
pycparser                                3.0
pycryptodome                             3.23.0
pydantic                                 2.13.4
pydantic_core                            2.46.4
pydantic-extra-types                     2.11.1
pydantic-settings                        2.14.1
pyelftools                               0.33
Pygments                                 2.20.0
PyJWT                                    2.12.1
PyPDF2                                   3.0.1
pypdfium2                                5.8.0
python-bidi                              0.6.10
python-dateutil                          2.9.0.post0
python-docx                              1.2.0
python-dotenv                            1.2.2
python-json-logger                       4.1.0
python-multipart                         0.0.28
PyYAML                                   6.0.2
pyzmq                                    27.1.0
quack-kernels                            0.4.1
ray                                      2.55.1
referencing                              0.37.0
regex                                    2026.5.9
requests                                 2.34.2
requests-toolbelt                        1.0.0
rich                                     15.0.0
rich-toolkit                             0.19.9
rignore                                  0.7.6
rpds-py                                  0.30.0
ruamel.yaml                              0.19.1
safetensors                              0.7.0
scikit-learn                             1.8.0
scipy                                    1.17.1
sentence-transformers                    5.5.0
sentencepiece                            0.2.1
sentry-sdk                               2.60.0
setproctitle                             1.3.7
setuptools                               79.0.1
shapely                                  2.1.2
shellingham                              1.5.4
six                                      1.17.0
sniffio                                  1.3.1
soundfile                                0.13.1
soxr                                     1.1.0
sse-starlette                            3.4.4
starlette                                0.52.1
supervisor                               4.3.0
sympy                                    1.14.0
tabulate                                 0.10.0
tenacity                                 9.1.4
threadpoolctl                            3.6.0
tiktoken                                 0.13.0
tilelang                                 0.1.9
tokenizers                               0.22.2
tokenspeed-mla                           0.1.2
tokenspeed-triton                        3.7.10.post20260505
torch                                    2.11.0+cu129
torch_c_dlpack_ext                       0.1.5
torchaudio                               2.11.0+cu129
torchvision                              0.26.0+cu129
tqdm                                     4.67.3
transformers                             5.8.1
triton                                   3.6.0
typer                                    0.25.1
typing_extensions                        4.15.0
typing-inspection                        0.4.2
ujson                                    5.12.1
urllib3                                  2.7.0
uuid_utils                               0.15.0
uvicorn                                  0.47.0
uvloop                                   0.22.1
vllm                                     0.22.0
watchfiles                               1.1.1
wcwidth                                  0.7.0
websockets                               16.0
wheel                                    0.46.3
wrapt                                    2.1.2
xformers                                 0.0.35
xgrammar                                 0.2.0
xxhash                                   3.7.0
yarl                                     1.23.0
z3-solver                                4.15.4.0
zipp                                     3.23.1
zstandard                                0.25.0
```
The transformers hints have disappeared, indicating that fast path packages should be working. However, with caching, Qwen 3.5's forward error is still significantly higher than other models.

@ArjunSrivastava1  I will try further testing using unsloth.

alright, thats good @Rocketknight1 was correct in his intuition for fast paths, i will look into if the docs need a note or rework of some sort, and dig in a bit in qwens implementation as to why caching is causing that forward error

The GDN implementation is correct. Two separate things to note:

- A real bug invalidated the float32 control: `dtype` is silently ignored when loading Qwen3.5 checkpoints through `AutoModelForCausalLM`, so the "float32" run was still bfloat16 (filed as #46459).
- The bf16 divergence itself is expected rounding, not an algorithmic difference between the chunk and recurrent paths.

All numbers below are from current main at effde20942, torch 2.11, CPU, no `causal-conv1d`/`fla`, @JWQZ's exact repro.

**1. The float32 control ran in bfloat16** 

After `AutoModelForCausalLM.from_pretrained("Qwen/Qwen3.5-0.8B", dtype=torch.float32)`, `next(model.parameters()).dtype` is `torch.bfloat16`. This is why fp32 looked identical to bf16: I get max=1.406250e-01, mean=1.903386e-02 for requested bf16, fp32, and fp64 alike.

**2. In true fp32 the divergence matches the Llama baseline** 

Casting after load (`.float()`) and rerunning Mode A vs Mode B:

```
max = 2.956390e-05, mean = 4.994844e-06   (Llama-3.2-1B baseline: 3.5e-5)
```

**3. The cached path computes the same function as the full forward**

- Kernels: with the internal fp32 casts of `torch_chunk_gated_delta_rule` / `torch_recurrent_gated_delta_rule` lifted to fp64, full-chunk vs pure recurrence vs chunked-prefill + recurrent decode with state handoff agree to ~9e-16 (prefill 10/64/70, T=83, mild and strong decay).
- Model: instrumenting layer 0 of Qwen3.5-0.8B, the decode token's post-conv q/k/v, g/beta, the `causal_conv1d_update` input, and the prefill-to-decode recurrent state are bit-identical between modes. The only difference is ~1.5e-5 in the kernel output (fp32-internal op ordering, rounded at bf16 io).

**4. Where 0.125 comes from** 

The per layer diffs are exact bf16 ULPs: layer 1 is 1.953125e-03 = 2^-9, and the top-5 logit diffs are all 0.125 = 2 ULPs at magnitudes 8-16. The ~1.5e-5 seed crosses bf16 rounding boundaries and re-quantizes at every layer, so the error grows in ULP steps through the 24 layers. This only exists in bf16, consistent with point 2.

The actionable bug is #46459. Once fixed, this issue can probably be closed as expected bf16 numerics. I can add a regression test pinning full-forward vs prefill+decode at a true-fp32 tolerance (~1e-4) if that is useful.


This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.