Add Doge model
Author: LoserCheems
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes #35889
Support the Doge-SLM family of small language models.


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

to: @ArthurZucker 

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc @MekkCyber

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Hi @LoserCheems, and thanks for the PR! The model looks cool and I like the paper too, but we're trying to add new models using **modular** in future. You can see a guide [here](https://huggingface.co/docs/transformers/main/en/modular_transformers), and an example modular PR [here](https://github.com/huggingface/transformers/pull/35569/files).

If you write `modular_doge.py` with inheritance then the `configuration_` and `modeling_` files will be auto-generated. This makes the PR much shorter and easier to review.

Thank you @Rocketknight1 , I've written `modular_doge.py`, but I'm sorry I don't quite understand **modular** and may make smoe mistakes...

Hi @losercheems yes, don't worry, it's a new feature so everyone is a bit confused about it! :sweat_smile: 

Your `modular_doge.py` file looks good! The next step is to find code that's copied from other models in `transformers`, and replace that with inheritance. This will make `modular_doge.py` much smaller, but the full `modeling_doge.py` will still be generated without inheritance. You can see some examples in the Qwen2.5 PR:

![image](https://github.com/user-attachments/assets/97ee6a9b-c8d6-449a-b64a-fc5080aa678c)

Classes like `DogeMLP` and `DogeForSequenceClassification` look like they use code from other library classes like Llama, so you could just inherit those instead in the modular file. You can run `make fix-copies` to regenerate `modeling_doge.py` and confirm that it still works.

Thank you @Rocketknight1. In fact, because the weight name or config name is different, can directly inherited class is not much, a total of ` RMSNorm `,  ` RotaryEmbedding ` , and ` DogeForSequenceClassification ` inherited from Llama.

Hi @losercheems, the last code quality error is caused by an unprotected `import torch`. These need to be guarded by `if is_torch_available` because some people have JAX-only or TF-only systems, and unguarded imports can make it impossible for them to use the library!

There are unrelated failing tests under `tests_torch` - you can ignore them for now, but once you can get code quality green then let me know and I'll review the PR.

Sorry @Rocketknight1, I mistakenly imported `PretrainedConfig` from `modeling_utils`, which is now fixed.

em🤓, There seems to be something wrong with `RotaryEmbedding` inherited from Llama.

gentle ping @Rocketknight1 

@Rocketknight1 code quality is already green!

Thank you @Rocketknight1 for your suggestion, I have finished the modification according to the suggestion!

Yes, looks good to me now! cc @cyrilvallez @ArthurZucker for core maintainer review. Failing tests are unrelated, should be fixed by rebasing after review / fixing nits

Thank you @ArthurZucker for your careful review. I have finished the modification according to your suggestion.

Cool having a look!

Thank you @Cyrilvallez! I am very sorry that I do not fully understand the weight mapping. Please help me convert the weights, and here are the model weights that can be referenced: https://huggingface.co/SmallDoge/Doge-160M-Instruct

In addition, there are still two nits:
- Custom flex_attention functions are not available in modular via `ALL_ATTENTION_FUNCTIONS.update({"flex_attention": flex_attention_forward})` updates to modeling
- Inheriting Llama in CausalLM causes the `wrong example model name` in the forward method.

@Cyrilvallez, I have finished the `convert_weights_to_hf` script.

@Cyrilvallez Request a review. There should be no issues other than the example error, but the impact should be minor as I found the same bug for all models using `modular`.

Thank you @Cyrilvallez for your careful review, I have fixed all issues as suggested.

@Cyrilvallez gentle ping, relevant tests have been completed.

Hey @Cyrilvallez, thank you very much for your long review. We've finally completed it!

Before mergin can you fix the test? 
You can inherit the test like it is done in Gemma3 test modeling

Hi @ArthurZucker, all the tests have been passed!

run-slow: doge

This comment contains run-slow, running the specified jobs: 

models: ['models/doge']
quantizations: [] ...

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, doge