Add ModernBERT Decoder Models - ModernBERT, but trained with CLM!
Author: orionw
Pull request: yes

This PR adds the new ModernBERTDecoder architecture: it's ModernBERT but trained from scratch as a CLM! This replaces my attempt in #35946 to fit them all in one class. Instead I'll have a script to convert them from encoders to decoders and vice versa by renaming the class and head name (`head` vs `lm_head`)

The models/paper are ready to go, but we wanted to have this up before we do the official release so we can have all the release artifacts ready. Thanks again for all your help on the last PR and for suggesting this approach!

FAQs:

Q: Why don't we just add this to ModernBERT?
A: Because we discussed in #35946 that it was too complicated to add the causal attention combined with everything else. This keeps it simpler and uses the standard attention functions

Q: How can BERT be a decoder? 
A: It's really the "ModernBERT" architecture trained from scratch via CLM. It's not MLM at all. We could rename the architecture to whatever we decide to name it in the paper, if that's preferable. But I thought this was the easiest to understand since they share the same everything else.

## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [x] Did you write any new necessary tests?


## Who can review?
@Rocketknight1 @gante @ArthurZucker 

Comments:
Hi @Rocketknight1 @gante, this is the new PR after our last conversation. I think it's much simpler as it's just a generic decoder with re-used ModernBERT parts. Thanks again for the help! 

cc @arthurzucker if you think this is appropriate for the library! I'm uncertain - I still think a smaller `ForCausalLM` wrapper that doesn't replace ModernBERT attention layers might be better, like `RobertaForCausalLM`.

Just to clarify though this does use normal causal attention like any other decoder, unlike ModernBERT (encoder). 

I'm a bit confused though, in the last PR ((https://github.com/huggingface/transformers/pull/35946) the attention layers were what was deemed too complicated (dealing with local vs global and causal vs non-casual at the same time). e.g. the last PR  was the `ForCausalLM` wrapper you mention, but it can't really be much "smaller" due to the complexity of handling that. 

EDIT, to quote the last comment from @Rocketknight1 in #35946

> Adding large tokenizer / internal layer changes to an MLM can create bugs, reduce performance and make it harder to maintain the core MLM functionality, which we don't want! Running MLMs as CLMs is a very niche thing to do.

We can't change ModernBERT to have a `ForCausalLM` without changing the internal attention layers. See @ArthurZucker's [comment on my last PR in the attention layer](https://github.com/huggingface/transformers/pull/35946/files#r2033255809) where he said to make a new class, which is this PR. 


Adding one more comment here to reinforce this - this isn't for a simple continued pre-train on top of ModernBERT. 

This is a **brand new from scratch decoder model suite (17m to 1B sizes) trained on 2T of open-data** but that uses the same architecture as ModernBERT in terms of shapes.  It is essentially a new decoder-LM for the library, it **also** just happens to be the same architecture which is great for comparisons. 

Super interested in this PR as well. Sending cheers to @orionw to boost your motivation. 
JB

Appreciate the support @jbdel! :) 

And thanks @ArthurZucker, sorry I didn't notice the new setup. It definitely looks a lot cleaner with those changes!

I got everything you suggested, so I think it's good to go! Please let me know if there's anything else.

Hi @ArthurZucker, just a ping if you have a sec :) 

Sorry, reviewing again!

Thanks @ArthurZucker, but unfortunately I don't see the review. Do you mind double checking? 

@ArthurZucker I merged in main but it seems like the recent lfm2 merge broke some things unrelated to this. Otherwise all the tests pass.

> FAILED tests/models/lfm2/test_modeling_lfm2.py::Lfm2ModelTest::test_sample_generate - AttributeError: 'Lfm2HybridConvCache' object has no attribute '_seen_tokens'

**[For maintainers]** Suggested jobs to run (before merge)

run-slow: auto, modernbert_decoder

Merged in the recent fixes from main, everything passes. Ready @ArthurZucker!

Thank you so much @ArthurZucker!! Appreciate your patience as I learn the new stuff :)