[new model] Add Zyphra/ZAYA1-8B
Author: JJJYmmm
Pull request: yes
Labels: name: New model

<!-- ci-dashboard-badge:start -->
[![CI](https://transformers-ci.lor-e.huggingface.cool/badge/pr?pr=45862)](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=45862)
<!-- ci-dashboard-badge:end -->

Zyphra recently released [ZAYA1-VL-8B](https://huggingface.co/Zyphra/ZAYA1-VL-8B), which has a small number of active parameters and looks like a nice fit here.

Since ZAYA1-VL depends on the text-only [ZAYA1-8B](https://huggingface.co/Zyphra/ZAYA1-8B) backbone, which has not been merged yet, this PR adds support for the text-only ZAYA1 backbone first. I can follow up with the VL model in a separate PR if preferred. 😃

I also noticed that #42669 worked on a similar integration. This PR updates the implementation to better fit the current v5 codebase, including a cleaner CCA cache design and other code cleanups. I also checked the numerical outputs against Zyphra's implementation: https://github.com/Zyphra/transformers/tree/zaya1 cc @nanduruganesh

Tests:

```bash
RUN_SLOW=1 python -m pytest tests/models/zaya/test_modeling_zaya.py -q

Comments:
Thank you very much for the rebase and cleanups! About the interleaved sliding window / rope theta confusion, these configs are in place for the ZAYA1-74B-preview model which also uses this branch but has 4k-SWA / 10k rope base every other layer. All of @vasqu's suggestions sound good to me, and another user has found additional fixes to support GRPO trainer on this branch ([PR](https://github.com/Zyphra/transformers/pull/2)). @JJJYmmm would you be able to integrate all the changes into your branch? Thanks again for this PR.

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_45862). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@vasqu thank you for the detailed review! it was really helpful for me to catch up with the latest changes and learn the unified code style, so that’s totally ok. thanks a lot for your time 😃

in the latest code, i fixed most of the inheritance issues. since the original checkpoint has some uncommon kwargs / weight layouts, which would require a lot of custom code, i wrote a conversion [script](https://github.com/JJJYmmm/transformers/blob/059912d60cf44859da49f08ad4eb0f2a02f46088/src/transformers/models/zaya/convert_zaya_weights_to_hf.py) and uploaded the converted 8b checkpoint here: [https://huggingface.co/JJJYmmm/ZAYA1-8B-HF](https://huggingface.co/JJJYmmm/ZAYA1-8B-HF). i also tested it with a fake 74b checkpoint with swa.

the conversion mainly does three things:

1. use more common names in the config, e.g. `intermediate_size`, and update some fields like `rope_parameters`
2. remove `nn.Sequential` and use explicit module names
3. combine the separate attention and mlp layers into a single `ZayaDecoderLayer`, with the corresponding config fixes, e.g. `num_hidden_layers: 80 -> 40`
4. 3d experts

what do you think about this conversion? 🫡 @nanduruganesh

> another user has found additional fixes to support GRPO trainer on this branch ([https://github.com/Zyphra/transformers/pull/2](https://github.com/Zyphra/transformers/pull/2))

@nanduruganesh i also checked this pr, and most of the fixes are already covered in the current branch. the only exception seems to be `8. router_aux_loss_coef`, but i think zaya does not use an auxiliary loss, right?

besides the conversion mentioned above, i also noticed a small detail in the official code about the SWA mask calculation. in the official branch, the [swa mask](https://github.com/Zyphra/transformers/blob/9e5966ce285b3d19035c344490d8c561f95a5952/src/transformers/models/zaya/modeling_zaya.py#L551-L560) is:

```python
if window_size > 0:
    causal_mask = (
        torch.ones((seq_length, seq_length), dtype=torch.bool, device=query_states.device)
        .tril_(diagonal=0)
        .triu_(diagonal=-window_size)
    )
    attn_weights.masked_fill_(~causal_mask, -1e4)
elif attention_mask is not None:  # no matter the length, we just slice it
    causal_mask = attention_mask[:, :, :, : key_states.shape[-2]]
    attn_weights = attn_weights + causal_mask
```

this means that in the swa branch, `attention_mask` is discarded. for now, i kept the [same behavior](https://github.com/JJJYmmm/transformers/blob/059912d60cf44859da49f08ad4eb0f2a02f46088/src/transformers/models/zaya/modeling_zaya.py#L880-L885) to preserve numerical consistency with the original implementation. is this expected?

EDIT: another reminder: in the conversion script, i increase swa `window_size` by one (`4096 -> 4097`).
this is because in the original logic, `.tril_(diagonal=0).triu_(diagonal=-window_size)` means the current query can attend to `window_size` keys. but in the current branch, `window_size` means the total window size directly.


Will try to take a look a bit later today @JJJYmmm, I'm out for the week after today - just wanted to notify so you arent surprised why Im suddenly not answering / responding

Edit: I think I responded to all things for now and a few comments are still left so waiting for now. TP seems to be the only somewhat harder case but waiting on Ferdinand for his opinion because it definitely won't be as straightforward as I expected (kv heads == 2 destroys a lot of assumptions :cry:)

Failing CI tests should be fixed on main

Sorry about spam pinging @nanduruganesh :pray: 
From the transformers side we are pretty much ready now, we only need the converted versions on the hub (and adjust the integration tests possibly, e.g. different model id). 

Thanks a lot on all this work @JJJYmmm :hugs: 

Thank you for all the hard work, my plan is to upload the converted version to the hub by the end of this week, though the current vLLM and llama.cpp implementations of our models depend on the old version of the checkpoint, so I will have to patch these first.

Sorry for the delay, I have updated the hub with the converted 8B/74B models @JJJYmmm @vasqu 

run-slow: zaya

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28224291701)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/zaya"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28224291701)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [f5295028](https://github.com/huggingface/transformers/commit/f5295028e1179f63c2727990ca2aa64d3930817c) | workflow commit (merge commit) |
| PR | [7ad188f8](https://github.com/JJJYmmm/transformers/commit/7ad188f8772c4486fa1d8efc8c247b10ebd7faa9) | branch commit (from PR) |
| main | [c96378c4](https://github.com/huggingface/transformers/commit/c96378c4136f5882fee50d8ff8ee1e9588a17eb6) | base commit (on `main`) |

### Model CI Report

❌ **[4 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/ce29fcc27d3d6742e3bda461f6d1a2d79acd2d7b/2026-06-26/runs/36921-28224291701/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [zaya](https://github.com/huggingface/transformers/actions/runs/28224291701/job/83612730567):
    tests/models/zaya/test_modeling_zaya.py::ZayaModelTest::test_torch_export (✅ ⟹ ❌)
    tests/models/zaya/test_modeling_zaya.py::ZayaIntegrationTest::test_model_cache_matches_full_forward (✅ ⟹ ❌)
    tests/models/zaya/test_modeling_zaya.py::ZayaIntegrationTest::test_model_generation (✅ ⟹ ❌)
    tests/models/zaya/test_modeling_zaya.py::ZayaIntegrationTest::test_model_logits (✅ ⟹ ❌)



@JJJYmmm we just merged the modeling refactor PR, whenever you have time to adjust, we can then merge afterwards :hugs: 

@vasqu Hi! I just updated the mask helper. If everything looks good, we can merge it. 🥂

run-slow: zaya

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28459558957)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/zaya"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28459558957)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [24aec84b](https://github.com/huggingface/transformers/commit/24aec84bcf62b96f4f217e0b139c4763a2454726) | workflow commit (merge commit) |
| PR | [20395cc3](https://github.com/JJJYmmm/transformers/commit/20395cc32088afd70319c44f4d08d71eb0f1001c) | branch commit (from PR) |
| main | [181beb3b](https://github.com/huggingface/transformers/commit/181beb3ba4c47098ed8cbc97ee250d1d45ae0107) | base commit (on `main`) |

### Model CI Report

❌ **[1 new failed tests from this PR](https://huggingface.co/datasets/hf-internal-testing/transformers_pr_ci/raw/92199705f14e89815e8d7f88600c2a20121b377d/2026-06-30/runs/37102-28459558957/ci_results_run_models_gpu/new_failures_with_bad_commit.json)** 😭

- [zaya](https://github.com/huggingface/transformers/actions/runs/28459558957/job/84344608777):
    tests/models/zaya/test_modeling_zaya.py::ZayaIntegrationTest::test_model_logits (✅ ⟹ ❌)



Should I run on our CI to update the values?

I just update the values 🫡

run-slow: zaya

[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28465614372)

This comment contains `run-slow`, running the specified jobs: 

models: ["models/zaya"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/28465614372)

### Commit Info
| Context | Commit | Description |
|---------|--------|-------------|
| RUN | [45454ee1](https://github.com/huggingface/transformers/commit/45454ee14870555861f1c4bff8c9ffbf566a078f) | workflow commit (merge commit) |
| PR | [5751cb07](https://github.com/JJJYmmm/transformers/commit/5751cb07f2b04016d78491e9ac4ad1c4e8a00150) | branch commit (from PR) |
| main | [181beb3b](https://github.com/huggingface/transformers/commit/181beb3ba4c47098ed8cbc97ee250d1d45ae0107) | base commit (on `main`) |

✅ No failing test specific to this PR 🎉 👏 !


Not sure what the tokenizer failure was about but a rerun helped and it passes now. Trying to merge now :saluting_face: 

CI is super weird, gonna check internally. Will merge as soon as I can (dont wanna force merge)

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, zaya

<!-- ci-dashboard-recap:start -->

---

### CI recap

**Dashboard:** [View test results in Grafana](https://transformers-ci.lor-e.huggingface.cool/d/pytest-observability-by-pr/pytest-observability-branch?var-pr=45862)
**Latest run:** [28523530075:1](https://github.com/huggingface/transformers/actions/runs/28524614250)
**Result:** `failure` | **Jobs:** 13 | **Tests:** 47,122 | **Failures:** 64 | **Duration:** 15h 24m

<!-- ci-dashboard-recap:end -->

Finally! :partying_face: