Adding support for OpenLMForCausalLM from DataComp
Author: EtashGuha
Pull request: yes

# What does this PR do?

Given the work from DataComp-LM: [https://arxiv.org/abs/2406.11794](url), this adds support for their architecture: OpenLMForCausulLM.

## Before submitting
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts, @qubvel
- speech models: @ylacombe, @eustlb
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Rocketknight1
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc
- chat templates: @Rocketknight1

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc

Documentation: @stevhliu

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
Minor typo, please correct: OpenLMForCausulLM --> OpenLMForCausalLM

Hi @EtashGuha, have you considered adding this model as a [custom code model](https://huggingface.co/docs/transformers/en/custom_models) instead? Lots of popular and widely used models are custom code models, and they can be loaded using `AutoModel` just like any other model!

In general, we only add models to the main `transformers` codebase once they get very very high usage, because the team at Hugging Face becomes responsible for maintaining them once they're in there. 

@Rocketknight1 thank you for pointing us into this - I missed this message yesterday

I will look into the solution above, thank you for your message!

> Hi @EtashGuha, have you considered adding this model as a [custom code model](https://huggingface.co/docs/transformers/en/custom_models) instead? Lots of popular and widely used models are custom code models, and they can be loaded using `AutoModel` just like any other model!
> 
> In general, we only add models to the main `transformers` codebase once they get very very high usage, because the team at Hugging Face becomes responsible for maintaining them once they're in there.

Would like to point out that DCLM is right now one of the strongest fully open-source models at 7B scale  - having fully open data dclm-baselines and open source training code. DLCM-7B is close to Llama 3 8B on benchs and is as far I can tell to  my knowledge the strongest open source model for now. I think HF might be interested in having it integrated in transformers codebase as strong fully open-source reference model, please consider.

@GeorgiosSmyrnis @jeniajitsev - as you can see, I was wrong! I think we can accept this as a PR to `transformers` as long as it's in the [modular](https://huggingface.co/docs/transformers/en/modular_transformers) style