

TITLE
Add the Nucleotide Transformer models

COMMENTS
6

REACTIONS
+1: 0
-1: 0
laugh: 0
hooray: 0
heart: 0
rocket: 0
eyes: 0

BODY
### Model description

The Nucleotide Transformer project is composed of four language models trained through Masked Language Modelling with an architecture similar to the ESM-1b architecture. The models have been developed by InstaDeep in collaboration with Nvidia and TUM. They have been trained on the NVIDIA Cambridge-1 cluster. The models include:

- one model with 500M parameters trained on the human reference genome
- one model with 500M parameters trained on a dataset composed of 3000+ human genomes
- one model with 2.5B parameters trained on a dataset composed of 3000+ human genomes
- one model with 2.5B parameters trained on a dataset composed of reference genomes from 850+ species

### Open source status

- [X] The model implementation is available
- [X] The model weights are available

### Provide useful links for the implementation

The model code and weights can be found here: https://github.com/instadeepai/nucleotide-transformer. The initial implementation is in Jax relying on haiku. 
The paper can be found here: https://www.biorxiv.org/content/10.1101/2023.01.11.523679v2.

I am one of the last authors of the paper. Other authors include @dallatt, @e-trop and @GRichard513.