Add HumanV: decoder-only causal LM
Author: humanprojectceo
Pull request: yes
Labels: name: New model

# What does this PR do?

This PR adds **HumanV**, a lightweight decoder-only causal language model architecture, to 🤗 Transformers.

Key additions:
- New `humanv` model implementation (`HumanVConfig`, `HumanVModel`, `HumanVForCausalLM`)
- RoPE-based causal self-attention, RMSNorm, and SiLU-based MLP blocks
- Registration so the model can be loaded via standard Transformers APIs (e.g. `AutoModelForCausalLM`) once merged

A small trained checkpoint is available on the Hugging Face Hub:
- https://huggingface.co/humanvprojectceo/nilla-story


<!-- Remove if not applicable -->
Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

Suggested reviewers for text model additions:
@ArthurZucker @Cyrilvallez


Comments:
View the CircleCI Test Summary for this PR:

https://huggingface.co/spaces/transformers-community/circle-ci-viz?pr=42982&sha=206397

This looks like it should be possible to implement under existing model architectures, right? Have you tried just converting the model checkpoints to `llama` or similar format instead of requiring a new architecture PR?

> This looks like it should be possible to implement under existing model architectures, right? Have you tried just converting the model checkpoints to `llama` or similar format instead of requiring a new architecture PR?

We agree that parts of HumanV overlap with existing decoder-only architectures (RoPE, RMSNorm, SiLU, and optional GQA/MHA). The motivation for a separate architecture PR is not to rebrand a LLaMA-like baseline, but to provide a stable upstream home for ongoing attention research we are actively implementing and benchmarking.

What is new / why not “just convert to LLaMA”?

HumanV is designed as a research-friendly attention sandbox within transformers, where attention patterns can be selected per-layer (e.g., full_attention vs sliding_attention), and where we can introduce non-standard attention backends without bending existing model classes beyond their intended scope.

We already implemented an opt-in Local–Global Block Sparse Attention mode (via config), and we are actively validating it for numerical stability (no NaNs) and correctness with small training runs (including TPU settings).

Sparse attention roadmap

Our goal is to integrate and evaluate modern sparse attention approaches (e.g., block-sparse / local-global hybrids).

The sparse path is fully behind configuration flags (e.g. use_sparse_attention=True, sparse_attention_impl="local_global_block"), so the default remains standard dense attention.

Training & reproducibility

We are currently training a fresh checkpoint using the new sparse attention algorithm, and once training is complete we will upload the resulting model checkpoint and update this PR with a direct Hugging Face link for reproducibility.

In short: this PR establishes the base HumanV architecture in transformers and enables a clean, opt-in path for sparse attention research that cannot be represented by a simple “convert to LLaMA” checkpoint when the attention mechanism itself is actively evolving.

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, humanv