Add gpt-sw3 model to transformers
Author: ekgren
Pull request: yes

This adds the gpt-sw3 models and tokenizer to hf. The models are developed by AI Sweden and others. They are gpt models trained from scratch with the nemo-megatron framework and will initially range in sizes from 128m to 20B. The models are multilingual and the languages in the models are English, Swedish, Norwegian, Danish and Icelandic.

Fixes # (issue) https://github.com/huggingface/transformers/issues/20176

@ArthurZucker 

Comments:
Hey! Feel free to ping me if you need any pointers! :) 
5seems like the history is a bit broken at this point `rebasing` with a force push should help. 

> Hey! Feel free to ping me if you need any pointers! :) 5seems like the history is a bit broken at this point `rebasing` with a force push should help.

Hi thank you for the help, did a rebase! 

We are soon going to put the models public on the hub, after that we hope that we are close to being able to make the PR ready for review.


_The documentation is not available anymore as the PR was closed or merged._

We have finally reached the point where most of the work seems to be done and would appreciate any feedback or a merge. The models haven't been made public yet but will be shortly. @ArthurZucker 

> Thanks for the great work! My biggest question here is : what are the architectural differences with `gpt2`? If there are none, I am not sure that we actually need to add any modelling files

So after reimplementing the code from nemo Megatron it turns out that with float 32 we get the exact same output between our implementation and the hf gpt-2 implementation and we reverted back to copying the GPT-2 hf implementation. So in the end we might not need it although we would of course be happy to have our model name featured as a class, the same as bloom and opt. The tokenizer on the other hand implements new behaviour.

What is your opinion on the model class?

@ArthurZucker just checking in that we adressed your comments! Thank you. We had a couple of questions otherwise it looks good from our end 🤩

Actually, it seems like the modeling code is exactly the same as for GPT2? In this case you can just set in the auto-mappings a correspondance `("gpt-sw3", "GPT2Model")` without needing to add a new model module.

Yep sorry for the late reply! Let's do the same as what was done with [BertJapanese](https://huggingface.co/docs/transformers/model_doc/bert-japanese). I'll review again sorry for not realising sooner `# Copied from` 😓 

> Actually, it seems like the modeling code is exactly the same as for GPT2? In this case you can just set in the auto-mappings a correspondance `("gpt-sw3", "GPT2Model")` without needing to add a new model module.

Thank you for your feedback, we're happy to follow your lead on how to proceed! So, if we understand you correctly, we should then remove `modeling_gpt_sw3.py`, `configuration_gpt_sw3.py` entirely?

> Yep sorry for the late reply! Let's do the same as what was done with [BertJapanese](https://huggingface.co/docs/transformers/model_doc/bert-japanese). I'll review again sorry for not realising sooner `# Copied from` sweat

Should we await further review or simply get started on this?

@sgugger @ArthurZucker 

Yes, that would be easier. Just remove the model and config files and in the auto mapping, use the GPT2 classes.

Thank you again for your help, I hope we have now resolved all of your issues. Do you see anything else required from our side in this PR? @sgugger @ArthurZucker 

A last nit @JoeyOhman , could you add an example of a pretrained model that you released being loaded in the doc? Like what is done with `BertJapanese` [here](https://moon-ci-docs.huggingface.co/docs/transformers/pr_20209/en/model_doc/bert-japanese). Would help people to understand that they can use the `GPT2` model with this tokenizer 😉 

Hey @ekgren could you add the correct checkpoints? They are probably private. 
See our CI fail [here](https://github.com/huggingface/transformers/actions/runs/3681682680/jobs/6228686391#step:8:1027) 

@sgugger @ArthurZucker Thank you for all the help and guidance! We have made all the tokenizers reffered to in the PR public.

We encountered some internal issues with the model sharing in the last minute, very sorry for that. Currently we are not allowed to share the model files publicly. However we can share the tokenizer and would very much like for it to be included in huggingface, since those with private access to the model easily can use the full hf ecosystem. We hope to be able to share the models fully public in the near future.

Hopefully our PR can still be included in the release now that the tests should pass.

No problem, I was thinking about the tokenizer rather than the actual checkpoints! You were mostly adding a tokenizer so I don't really see an issue with this 😉 Thanks for the contribution!