[MODEL] Nanochat implementation
Author: burtenshaw
Pull request: yes

This PR introduces Andrej Karpathy's `nanochat` to transformers.

The transformers compatible weights are currently available in two places:

- A [pr](https://huggingface.co/karpathy/nanochat-d32/discussions/1) on Karpathy's repo (large model)
- A [repo](https://huggingface.co/nanochat-students/nanochat-d20) on the community org  (small model)

# ToDo

- CI is failing dues to tests for `SwiftFormerForQuestionAnswering` etc.


Comments:
The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_41634). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

> An initial review, I went a bit into the deep end with the RoPE but I hope I could explain what's going on 😅
> 
> The biggest tasks are
> 
> * modular
> * RoPE
> * `_can_record` to properly support some features

@vasqu Thanks for the thorough review. It's really helpful to me. I've now responded to all the comments in line, reran the conversion, and updated the model repos. 

edit: I ended up having to revert some of last changes and (re)use less llama modules because the model performance regressed. https://github.com/huggingface/transformers/pull/41634/commits/99801d04cd9cae2c9b614708f11444e290fb6ac1

@vasqu Thanks again. It's now running on a llama rope with just a sine inversion. https://github.com/huggingface/transformers/pull/41634/commits/57e0e3573630200c551af59fe7bdb0c27ffe56a2

@vasqu Thanks again. I've integrated all these changes and there are no regression. Just one [nit](https://github.com/huggingface/transformers/pull/41634#discussion_r2445801535) I couldn't understand.

Thanks again @vasqu ! I've responded to the inline TODOs. Two points:

- I definitely think we should stick with `karpathy/nanochat-d32` because it's his model and has more visibility. He's agreed to merge the weights via a hub PR.
- I changed the conversion script to work with the output of the `nanochat` repos configuration. This should be more compatible.

run-slow: gemma2, gemma3, gemma3n, vaultgemma

This comment contains run-slow, running the specified jobs: 

models: ['models/gemma2', 'models/gemma3', 'models/gemma3n', 'models/nanochat', 'models/vaultgemma']
quantizations: [] ...

run-slow: vaultgemma

This comment contains run-slow, running the specified jobs: 

models: ['models/nanochat', 'models/vaultgemma']
quantizations: [] ...

run-slow: gemma3

This comment contains run-slow, running the specified jobs: 

models: ['models/gemma3', 'models/nanochat']
quantizations: [] ...

run-slow: gemma3

This comment contains run-slow, running the specified jobs: 

models: ['models/gemma3', 'models/nanochat']
quantizations: [] ...

run-slow: nanochat

This comment contains run-slow, running the specified jobs: 

models: ['models/nanochat']
quantizations: [] ...

run-slow: nanochat

This comment contains run-slow, running the specified jobs: 

models: ['models/nanochat']
quantizations: [] ...

@ArthurZucker for core maintainer review when you are back :D 

looking forward to the review. 

@vasqu I fixed the nits in configs. Can you rerun `slow` to be sure.

run-slow: nanochat

This comment contains run-slow, running the specified jobs: 

models: ['models/nanochat']
quantizations: [] ...

run-slow: nanochat

This comment contains run-slow, running the specified jobs: 

models: ['models/nanochat']
quantizations: [] ...

run-slow: nanochat

This comment contains `run-slow`, running the specified jobs: 

models: ["models/nanochat"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19009944536)

✅ No failing test specific to this PR 🎉 !


run-slow: nanochat

This comment contains `run-slow`, running the specified jobs: 

models: ["models/nanochat"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19016165856)

✅ No failing test specific to this PR 🎉 !


cc @ArthurZucker @Cyrilvallez for core maintainer when you have time

Have to resolve the conflicts :-)

Not anymore @ydshieh :p

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, gemma2, gemma3, gemma3n, nanochat, vaultgemma

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, gemma2, gemma3, gemma3n, nanochat, vaultgemma

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, gemma2, gemma3, gemma3n, nanochat, vaultgemma

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, gemma2, gemma3, gemma3n, nanochat, vaultgemma

run-slow: nanochat

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, gemma2, gemma3, gemma3n, nanochat, vaultgemma

This comment contains `run-slow`, running the specified jobs: 

models: ["models/nanochat"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19639054028)

✅ No failing test specific to this PR 🎉 !


And super sorry for the delay on this! Past 2 weeks have been really crazy! 😬

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, gemma2, gemma3, gemma3n, nanochat, vaultgemma

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, gemma2, gemma3, gemma3n, nanochat, vaultgemma

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, gemma2, gemma3, gemma3n, nanochat, vaultgemma

> Amazing work! Super clean, and you also refactored gemmas super nicely 🤗🤗🚀 It's really almost ready to be merged, only 2 frictions points are the `_init_weights`, and the tp_plan currently! They need to be changed a bit for correctness/efficiency! Let me know if something is unclear about that!

@Cyrilvallez Thanks for the review! I've responded to all those changes and added suggestions.

run-slow: nanochat

This comment contains `run-slow`, running the specified jobs: 

models: ["models/nanochat"]
quantizations: []

## CI Results
[Workflow Run ⚙️](https://github.com/huggingface/transformers/actions/runs/19675333954)

✅ No failing test specific to this PR 🎉 !


**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, gemma2, gemma3, gemma3n, nanochat, vaultgemma

@Cyrilvallez @vasqu Thanks both. Looks like tests are good and I've merged main. When do you think this can be merged?

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, gemma2, gemma3, gemma3n, nanochat, vaultgemma

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, gemma2, gemma3, gemma3n, nanochat, vaultgemma

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, gemma2, gemma3, gemma3n, nanochat, vaultgemma

Thanks again @Cyrilvallez @vasqu . It was a pleasure working with you. 

Likewise @burtenshaw :)