Add Nemotron HF Support
Author: suiyoubi
Pull request: yes

# What does this PR do?

<!--
Congratulations! You've made it this far! You're not quite done yet though.

Once merged, your PR is going to appear in the release notes with the title you set, so make sure it's a great title that fully reflects the extent of your awesome contribution.

Then, please replace this with a description of the change and which issue is fixed (if applicable). Please also include relevant motivation and context. List any dependencies (if any) that are required for this change.

Once you're done, someone will review your PR shortly (see the section "Who can review?" below to tag some potential reviewers). They may suggest changes to make the code even better. If no one reviewed your PR after a week has passed, don't hesitate to post a new comment @-mentioning the same persons---sometimes notifications get lost.
-->

<!-- Remove if not applicable -->

Fixes # (issue)


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [ ] Did you read the [contributor guideline](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md#create-a-pull-request),
      Pull Request section?
- [ ] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [ ] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/transformers/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/transformers/tree/main/docs#writing-source-documentation).
- [ ] Did you write any new necessary tests?


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

<!-- Your PR will be replied to more quickly if you can figure out the right person to tag with @

 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- text models: @ArthurZucker
- vision models: @amyeroberts
- speech models: @sanchit-gandhi
- graph models: @clefourrier

Library:

- flax: @sanchit-gandhi
- generate: @zucchini-nlp (visual-language models) or @gante (all others)
- pipelines: @Narsil
- tensorflow: @gante and @Rocketknight1
- tokenizers: @ArthurZucker
- trainer: @muellerzr and @SunMarc

Integrations:

- deepspeed: HF Trainer/Accelerate: @muellerzr
- ray/raytune: @richardliaw, @amogkam
- Big Model Inference: @SunMarc
- quantization (bitsandbytes, autogpt): @SunMarc 

Documentation: @stevhliu and @MKhalusova

HF projects:

- accelerate: [different repo](https://github.com/huggingface/accelerate)
- datasets: [different repo](https://github.com/huggingface/datasets)
- diffusers: [different repo](https://github.com/huggingface/diffusers)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Maintained examples (not research project or legacy):

- Flax: @sanchit-gandhi
- PyTorch: See Models above and tag the person corresponding to the modality of the example.
- TensorFlow: @Rocketknight1

 -->


Comments:
cc @ArthurZucker 

this PR is essential for easy support of recently open-sourced Minitron-4B and Minitron-8B. 

Hi @ArthurZucker,

Thank you for your review and support. I'm a colleague of @suiyoubi.

Our colleague @gheinrich has addressed your comments and created a PR to @suiyoubi's forked branch. 
- https://github.com/suiyoubi/transformers/pull/3

However, the issue is that @suiyoubi is not currently available and may not be able to be online sometime soon. That means, @gheinrich's [PR](https://github.com/suiyoubi/transformers/pull/3) cannot be merged into [suiyoubi:aot/nemotron-support](https://github.com/suiyoubi/transformers/tree/aot/nemotron-support).

This PR is a very important and urgent for us as it blocks recently open-sourced Nemotron models. Would it be possible to merge @gheinrich's PR into this PR and merge into the main branch?

I'm not sure if it is possible to switch the source branch in an existing PR. One option is to create a new branch (e.g., `aot/nemotron-support`) in this repo by simply copying @suiyoubi's [suiyoubi:aot/nemotron-support](https://github.com/suiyoubi/transformers/tree/aot/nemotron-support) and then @gheinrich will change the target branch to that branch. 

Any other suggestions/workarounds would also be much appreciated.

Thank you for your support!

cc: @suiyoubi @gheinrich

I think @suiyoubi can open a PR to transformers directly as long as his repo is a fork. Do whatever works, I am happy to review. You just need a fork of transformers! 

Ping me once you have any PR ready I'll review asap to make sure we get this shipped ! 

Thank you @ArthurZucker! A tricky thing is that @suiyoubi is not currently available online and will not be anytime soon. The latest changes are made by another member, who does not have write permission to @suiyoubi's forked repo.

Do you think only the option is to create a new PR (and mention this PR to link it to the new PR)?

Hi @ArthurZucker, 

~~I created a new PR (#32254) on behalf of @suiyoubi and @gheinrich. Please review the new PR. Thank you!~~
Closed the PR (#32254) as @suiyoubi is back and has made updates on this original PR. Please continue reviewing this original ticket. Thanks!

Hi @ArthurZucker ,

I updated the PR with the changes @suhara  and @gheinrich had before.
Let me know how you want to proceed. Thanks! 

Hi @ArthurZucker,

Can you take a look at the PR when you get a chance? Thank you for your support!

For sure

Could you fix the `ERROR tests/utils/test_file_utils.py - AttributeError: module transformers.models.nemotron has no attribute NemotronForPreTraining` failing tests as well? 🤗 

@ArthurZucker  I have some question relating to the `Copied from` convention, if the code is not exactly the same but adapted from it, should I still have the `Copied from` comment?
i.e. For the nemotron rope, I need to add the two lines: 
```
            if partial_rotary_factor < 1.0:
                dim = int(dim * partial_rotary_factor)
```
Does this still valid for `copied from` ? I see the `check_copies.py` is failing due to this

If it is adapted from you can remove the copied from of the class (unless a function alone is different, in that case, you can add # Ignore Copy to the function that is different!

Thanks a LOT for your hard work, almost finished! 

@ArthurZucker  All the issues has been resolved. Please take a look again! Thanks a lot for the review! Looks like there is still some CI not passed. I'll address them today.

I have one more question, for now we use `LlamaTokenizer` for all Nemotron Tokenizer, and I don't know if this would cause the confusing in the actual release (i.e. in all nemotron models, the tokenizer config would always point to llama). Is there a good way to without copying the llamatokenizer code ? Is there anyway in hf's transformers that allow "alias" feature for this use case ?

IMO the best thing is to use `PreTokenizerFast` 

> IMO the best thing is to use `PreTokenizerFast`

But `PreTrainedTokenizerFast` does not support directly load setencepiece (`tokenizer.model`) file.



> But PreTrainedTokenizerFast does not support directly load setencepiece (tokenizer.model) file.

It does not, but we can simply convert it in the conversion script! It's really up to you 🤗 

> > But PreTrainedTokenizerFast does not support directly load setencepiece (tokenizer.model) file.
> 
> It does not, but we can simply convert it in the conversion script! It's really up to you 🤗

Could you please show me how to convert the `tokenizer.model` into the `PretrainedTokenizerFast` ? If it is possible, I would like to add this to the conversion script.

failing doc:
```
- model_doc/nemotron
Add them to ../transformers/docs/source/en/_toctree.yml.
```
and we can merge!

The docs for this PR live [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_31699). All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Thank you @ArthurZucker! We appreciate your support!