Add AevaModel
Author: louzongzhi
Pull request: yes

# What does this PR do?

This PR introduces the **Aeva** model implementation to the Transformers library—a next-generation large language model (LLM) architecture built on three core innovations designed to push the boundaries of efficiency, scalability, and representation quality in modern language modeling.

The implementation includes three fundamental architectural breakthroughs specifically engineered for LLMs:

1. **Hierarchical Sparse MoE (HGSCMoE)**: A brain-inspired, event-driven routing mechanism that leverages temporal memory and threshold-based sparsity for expert selection. It implements hierarchical expert grouping for balanced load distribution, shared expert capacity, and anchor experts to stabilize training. The advanced load balancing loss with group-aware routing optimization ensures efficient utilization of expert capacity across massive-scale deployments.

2. **Hybrid Suffix Attention (with ROSA)**: A novel attention paradigm that intelligently combines sliding window attention for local context modeling with linear attention for scalable long-context processing. The integration of Rapid Online Suffix Automaton (ROSA) provides ultra-fast autoregressive decoding using suffix automaton theory with statistical fallback, delivering state-of-the-art inference speeds while maintaining generation quality through layer-wise attention type scheduling.

3. **Deep-Embedded MHC (Manifold-Constrained Hyper-Connections)**: Enhances representation learning through manifold-constrained transformations that stabilize high-dimensional feature learning in ultra-deep networks. It features deep embedding tables directly integrated into attention computations for keys and values, gated control mechanisms with learnable channel-wise scaling, and Sinkhorn-Knopp normalization for stable feature aggregation across layers.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
  [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

This PR introduces a novel LLM architecture with advanced attention mechanisms and MoE routing. Key reviewers should include:

- Text models: @ArthurZucker @Cyrilvallez
- Generate/Generation: @gante  
- Attention mechanisms: @vasqu @ArthurZucker @CyrilVallez

Feel free to tag additional reviewers interested in large-scale model architectures, MoE systems, or efficient attention mechanisms.

Comments:
**[For maintainers]** Suggested jobs to run (before merge)

run-slow: aeva, auto

View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=43397&sha=274bba

Hi @louzongzhi, are there any pre-trained checkpoints for this model? We generally don't add architectures until a model checkpoint exists that uses them!

> Hi @louzongzhi, are there any pre-trained checkpoints for this model? We generally don't add architectures until a model checkpoint exists that uses them!

Understood. I originally planned to add the architecture first and then train, but I will proceed to upload directly to the hub instead.

No probs! I liked the manifold-constrained hyperconnections paper a lot, so I'm excited to see a model using them. You can make it a [custom code model](https://huggingface.co/docs/transformers/en/custom_models) without needing any PR to Transformers - we generally only merge models into Transformers when they have SOTA performance or a lot of users, because once they're in the main library then the Transformers team is responsible for maintaining the code.