Mamba mismatched sizes
Author: joelburget
Pull request: no
Labels: name: bug

### System Info

- `transformers` version: 4.45.1
- Platform: Linux-6.8.0-40-generic-x86_64-with-glibc2.35
- Python version: 3.10.12
- Huggingface_hub version: 0.25.1
- Safetensors version: 0.4.5
- Accelerate version: 0.34.2
- Accelerate config: 	not found
- PyTorch version (GPU?): 2.2.0+cu121 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using distributed or parallel set-up in script?: <fill in>
- Using GPU in script?: Yes
- GPU type: NVIDIA A100-SXM4-80GB

### Who can help?

@ArthurZucker 

### Information

- [ ] The official example scripts
- [ ] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [ ] My own task or dataset (give details below)

### Reproduction

```
from transformers import AutoModelForCausalLM, MambaForCausalLM
model_name = "state-spaces/mamba-2.8b"
model = AutoModelForCausalLM.from_pretrained(model_name)  # or MambaForCausalLM.from_pretrained
```

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 4014, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 4559, in _load_pretrained_model
    raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
RuntimeError: Error(s) in loading state_dict for MambaForCausalLM:
	size mismatch for backbone.embeddings.weight: copying a param with shape torch.Size([50280, 2560]) from checkpoint, the shape in current model is torch.Size([50277, 768]).
	size mismatch for backbone.layers.0.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
```

<details>
<summary>Full error message</summary>

```
>>> model = AutoModelForCausalLM.from_pretrained(model_name)
The fast path is not available because one of `(selective_state_update, selective_scan_fn, causal_conv1d_fn, causal_conv1d_update, mamba_inner_fn)` is None. Falling back to the sequential implementation of Mamba, as use_mambapy is set to False. To install follow https://github.com/state-spaces/mamba/#installation and https://github.com/Dao-AILab/causal-conv1d. For the mamba.py backend, follow https://github.com/alxndrTL/mamba.py.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/dist-packages/transformers/models/auto/auto_factory.py", line 564, in from_pretrained
    return model_class.from_pretrained(
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 4014, in from_pretrained
    ) = cls._load_pretrained_model(
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 4559, in _load_pretrained_model
    raise RuntimeError(f"Error(s) in loading state_dict for {model.__class__.__name__}:\n\t{error_msg}")
RuntimeError: Error(s) in loading state_dict for MambaForCausalLM:
	size mismatch for backbone.embeddings.weight: copying a param with shape torch.Size([50280, 2560]) from checkpoint, the shape in current model is torch.Size([50277, 768]).
	size mismatch for backbone.layers.0.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.0.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.0.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.0.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.0.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.0.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.0.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.0.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.0.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.0.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.1.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.1.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.1.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.1.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.1.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.1.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.1.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.1.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.1.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.1.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.2.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.2.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.2.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.2.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.2.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.2.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.2.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.2.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.2.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.2.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.3.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.3.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.3.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.3.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.3.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.3.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.3.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.3.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.3.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.3.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.4.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.4.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.4.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.4.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.4.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.4.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.4.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.4.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.4.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.4.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.5.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.5.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.5.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.5.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.5.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.5.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.5.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.5.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.5.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.5.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.6.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.6.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.6.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.6.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.6.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.6.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.6.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.6.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.6.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.6.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.7.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.7.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.7.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.7.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.7.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.7.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.7.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.7.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.7.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.7.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.8.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.8.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.8.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.8.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.8.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.8.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.8.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.8.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.8.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.8.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.9.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.9.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.9.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.9.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.9.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.9.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.9.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.9.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.9.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.9.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.10.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.10.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.10.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.10.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.10.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.10.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.10.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.10.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.10.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.10.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.11.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.11.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.11.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.11.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.11.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.11.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.11.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.11.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.11.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.11.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.12.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.12.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.12.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.12.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.12.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.12.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.12.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.12.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.12.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.12.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.13.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.13.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.13.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.13.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.13.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.13.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.13.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.13.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.13.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.13.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.14.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.14.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.14.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.14.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.14.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.14.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.14.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.14.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.14.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.14.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.15.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.15.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.15.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.15.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.15.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.15.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.15.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.15.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.15.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.15.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.16.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.16.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.16.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.16.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.16.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.16.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.16.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.16.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.16.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.16.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.17.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.17.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.17.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.17.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.17.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.17.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.17.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.17.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.17.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.17.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.18.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.18.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.18.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.18.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.18.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.18.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.18.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.18.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.18.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.18.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.19.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.19.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.19.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.19.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.19.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.19.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.19.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.19.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.19.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.19.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.20.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.20.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.20.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.20.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.20.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.20.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.20.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.20.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.20.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.20.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.21.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.21.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.21.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.21.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.21.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.21.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.21.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.21.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.21.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.21.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.22.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.22.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.22.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.22.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.22.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.22.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.22.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.22.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.22.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.22.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.23.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.23.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.23.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.23.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.23.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.23.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.23.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.23.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.23.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.23.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.24.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.24.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.24.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.24.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.24.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.24.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.24.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.24.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.24.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.24.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.25.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.25.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.25.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.25.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.25.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.25.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.25.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.25.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.25.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.25.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.26.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.26.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.26.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.26.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.26.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.26.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.26.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.26.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.26.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.26.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.27.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.27.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.27.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.27.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.27.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.27.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.27.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.27.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.27.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.27.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.28.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.28.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.28.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.28.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.28.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.28.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.28.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.28.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.28.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.28.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.29.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.29.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.29.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.29.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.29.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.29.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.29.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.29.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.29.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.29.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.30.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.30.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.30.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.30.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.30.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.30.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.30.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.30.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.30.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.30.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.layers.31.norm.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	size mismatch for backbone.layers.31.mixer.A_log: copying a param with shape torch.Size([5120, 16]) from checkpoint, the shape in current model is torch.Size([1536, 16]).
	size mismatch for backbone.layers.31.mixer.D: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.31.mixer.conv1d.weight: copying a param with shape torch.Size([5120, 1, 4]) from checkpoint, the shape in current model is torch.Size([1536, 1, 4]).
	size mismatch for backbone.layers.31.mixer.conv1d.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.31.mixer.in_proj.weight: copying a param with shape torch.Size([10240, 2560]) from checkpoint, the shape in current model is torch.Size([3072, 768]).
	size mismatch for backbone.layers.31.mixer.x_proj.weight: copying a param with shape torch.Size([192, 5120]) from checkpoint, the shape in current model is torch.Size([80, 1536]).
	size mismatch for backbone.layers.31.mixer.dt_proj.weight: copying a param with shape torch.Size([5120, 160]) from checkpoint, the shape in current model is torch.Size([1536, 48]).
	size mismatch for backbone.layers.31.mixer.dt_proj.bias: copying a param with shape torch.Size([5120]) from checkpoint, the shape in current model is torch.Size([1536]).
	size mismatch for backbone.layers.31.mixer.out_proj.weight: copying a param with shape torch.Size([2560, 5120]) from checkpoint, the shape in current model is torch.Size([768, 1536]).
	size mismatch for backbone.norm_f.weight: copying a param with shape torch.Size([2560]) from checkpoint, the shape in current model is torch.Size([768]).
	You may consider adding `ignore_mismatched_sizes=True` in the model `from_pretrained` method.
```

</details>

### Expected behavior

The model to load without error

Comments:
You need to add a "-hf" at the end, i.e. `model_name="state-spaces/mamba-2.8b-hf"`. 

Thanks, I think your comment got me unstuck! (Though I do think it should fail in a clearer way.)

The problem is that "state-spaces/mamba-2.8b" are the weights of the original paper with the repo at https://github.com/state-spaces/mamba while "state-spaces/mamba-2.8b-hf" are the converted weights etc. which is needed to be compatible with transformers.

It is confusing tho, especially with the naming!