mamba2 mixer different outputs between fast, slow, training, and inference paths.
Author: tsengalb99
Pull request: no
Labels: name: bug

### System Info

```
- huggingface_hub version: 0.31.4
- Platform: Linux-5.15.0-140-generic-x86_64-with-glibc2.35
- Python version: 3.13.0
- Running in iPython ?: No
- Running in notebook ?: No
- Running in Google Colab ?: No
- Running in Google Colab Enterprise ?: No
- Token path ?: /home/alberttseng/.cache/huggingface/token
- Has saved token ?: True
- Who am I ?: at676
- Configured git credential helpers: 
- FastAI: N/A
- Tensorflow: N/A
- Torch: 2.9.0.dev20250803+cu129
- Jinja2: 3.1.6
- Graphviz: N/A
- keras: N/A
- Pydot: N/A
- Pillow: 11.2.1
- hf_transfer: N/A
- gradio: N/A
- tensorboard: N/A
- numpy: 2.2.6
- pydantic: 2.10.3
- aiohttp: 3.11.18
- hf_xet: N/A
- ENDPOINT: https://huggingface.co
- HF_HUB_CACHE: /home/alberttseng/.cache/huggingface/hub
- HF_ASSETS_CACHE: /home/alberttseng/.cache/huggingface/assets
- HF_TOKEN_PATH: /home/alberttseng/.cache/huggingface/token
- HF_STORED_TOKENS_PATH: /home/alberttseng/.cache/huggingface/stored_tokens
- HF_HUB_OFFLINE: False
- HF_HUB_DISABLE_TELEMETRY: False
- HF_HUB_DISABLE_PROGRESS_BARS: None
- HF_HUB_DISABLE_SYMLINKS_WARNING: False
- HF_HUB_DISABLE_EXPERIMENTAL_WARNING: False
- HF_HUB_DISABLE_IMPLICIT_TOKEN: False
- HF_HUB_ENABLE_HF_TRANSFER: False
- HF_HUB_ETAG_TIMEOUT: 10
- HF_HUB_DOWNLOAD_TIMEOUT: 10

```

### Who can help?

@ArthurZucker @Cyrilvallez 

### Information

- [ ] The official example scripts
- [x] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [x] My own task or dataset (give details below)

### Reproduction

In https://github.com/huggingface/transformers/blob/0720e206c6ba28887e4d60ef60a6a089f6c1cc76/src/transformers/models/mamba2/modeling_mamba2.py#L656, when given the same input to `hidden_states` and setting the other arguments to `None`, I get different results for the fast path (`self.cuda_kernels_forward`) with `self.training = True`, the fast path with `self.training = False`, and the slow path. 

### Expected behavior

These should all be equivalent.

Comments:
Hey @tsengalb99,

what model are you using to check? The bigger the model the easier it is to get discrepancies as different (non) kernels are involved. 

The slow path is known to have a larger difference especially in lower precision contexts, any improvements there are appreciated. For the training vs inference path (in fast path), this is likely due to having a completely fused kernel `mamba_split_conv1d_scan_combined` vs. applying 2 kernels one after the other, i.e. `causal_conv1d_fn` then `mamba_chunk_scan_combined`. We strictly follow the original codebase at https://github.com/state-spaces/mamba/tree/main for applying the kernels. So if this issue appears here, then it should also appear there as well, i.e. it's hard to solve on our side.

Tl;dr: Fast Path - different kernel applications can easily introduce fluctuations and would likely appear in the original codebase as well. Slow Path - known to be less precise and is more optimized towards speed than accuracy.

I am trying to train my own model that uses a single Mamba2 decoder layer inside of it. I'm getting a very large discrepancy (not numerical error, but >>100% error), so it seems like there is a significant implementation difference between the different paths.

Can you share your config for the model / a script for reproducibility? Imo, it is still not easy to say whether this is an issue in the orginal kernels rather than the usage of them. 

The backward pass in the mamba kernels is non-deterministic just fyi. Possible relevant issues 
- https://github.com/state-spaces/mamba/issues/548
- https://github.com/state-spaces/mamba/issues/545
- https://github.com/state-spaces/mamba/issues/75

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.