Add the Nucleotide Transformer models
Author: ranzenTom
Pull request: no
Labels: name: New model

### Model description

The Nucleotide Transformer project is composed of four language models trained through Masked Language Modelling with an architecture similar to the ESM-1b architecture. The models have been developed by InstaDeep in collaboration with Nvidia and TUM. They have been trained on the NVIDIA Cambridge-1 cluster. The models include:

- one model with 500M parameters trained on the human reference genome
- one model with 500M parameters trained on a dataset composed of 3000+ human genomes
- one model with 2.5B parameters trained on a dataset composed of 3000+ human genomes
- one model with 2.5B parameters trained on a dataset composed of reference genomes from 850+ species

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

The model code and weights can be found here: https://github.com/instadeepai/nucleotide-transformer. The initial implementation is in Jax relying on haiku. 
The paper can be found here: https://www.biorxiv.org/content/10.1101/2023.01.11.523679v2.

I am one of the last authors of the paper. Other authors include @dallatt, @e-trop and @GRichard513.

Comments:
Question for the huggingface team:

The Nucleotide Transformer networks use the ESM-1b architecture and ESM is supported in transformers, however, the ESM implementation relies on PyTorch whereas the Nucleotide Tansformer are implemented in Jax. Would you recommend to port them in PyTorch by following the ESM code or to port them in Jax directly? Also, if we were to port in Jax; Do you plan to support [haiku](https://github.com/deepmind/dm-haiku) which has been used for the original implementation?  

Hi @ranzenTom! If your architecture is **exactly** the same as an existing model in `transformers` like ESM-1b or ESM-2, the quickest approach would be to use our existing ESM-1b implementation in PyTorch or TF and crossload your weights into it, then upload that as a model checkpoint. We could then optionally add a JAX/FLAX port of the model architecture in a later PR, but this would get your model into `transformers` with the minimum of fuss.

The way models are stored in `transformers` is that the main repository contains the architectural code, and weight checkpoints and model configurations are stored in user repos. As such, if your model deviates from the ESM-1b architecture in any way, we won't be able to use this approach, because the model code in the main repo won't run your weights correctly. 

If you're confident that the architectures are the same and you want to try porting the weights yourself, I made a quick [demo notebook](https://colab.research.google.com/drive/1I6uo8SPAnikcOiMQY-2eyofZo3DAbdA-?usp=sharing) to show the process.

If there are any architectural differences, we should still be able to port Nucleotide Transformer! We'll just have to use a different method, and not just treat it as an ESM-1b checkpoint.

Also, I should mention - if you have any questions or need any assistance at any point here, please let us know! We're seeing a lot of interest in biological and clinical models, with models like ESM and BioClinicalBERT getting hundreds of thousands of downloads per month, so we're excited for you to be our first DNA models too!

For anyone following this issue: we have started a slack channel to further iterate on this. :) 

Hi, I'm interested in developing/using nucleotide levels LLMS for variant calling applications. Please let me know how I can be added to the Slack channel on the integration of nucleotide transformer within hugging face transformer.

They've just been added, and so we're going to close this issue!

You can see the model list [here](https://huggingface.co/InstaDeepAI). You can use them using standard classes like `AutoTokenizer`,`AutoModelForMaskedLM` or `AutoModelForSequenceClassification`. We fixed a couple of issues related to this in recent PRs, so please update transformers to the latest version on main with `pip install --upgrade git+https://github.com/huggingface/transformers.git` before trying to use them.

PyTorch only for now, but we expect TensorFlow versions to be available in a couple of days!