docs: add finetuned models to readme

579d5b0 verified 3 months ago

No virus

4.7 kB

	---
	tags:
	- dna
	- human_genome
	---

	# GENA-LM (gena-lm-bert-large-t2t)

	GENA-LM is a Family of Open-Source Foundational Models for Long DNA Sequences.

	GENA-LM models are transformer masked language models trained on human DNA sequence.

	Differences between GENA-LM (`gena-lm-bert-large-t2t`) and DNABERT:
	- BPE tokenization instead of k-mers;
	- input sequence size is about 4500 nucleotides (512 BPE tokens) compared to 512 nucleotides of DNABERT
	- pre-training on T2T vs. GRCh38.p13 human genome assembly.

	Source code and data: https://github.com/AIRI-Institute/GENA_LM

	Paper: https://www.biorxiv.org/content/10.1101/2023.06.12.544594

	This repository also contains models that are finetuned on downstream tasks:
	- promoters prediction 300bp (branch [promoters_300_run_1](https://huggingface.co/AIRI-Institute/gena-lm-bert-large-t2t/tree/promoters_300_run_1))
	- promoters prediction 2000bp (branch [promoters_2000_run_1](https://huggingface.co/AIRI-Institute/gena-lm-bert-large-t2t/tree/promoters_2000_run_1))
	- splice sites prediction (branch [spliceai_run_1](https://huggingface.co/AIRI-Institute/gena-lm-bert-large-t2t/tree/spliceai_run_1))

	and models that are used in our [GENA-Web](https://dnalm.airi.net) web tool for genomic sequence annotation:
	- promoters prediction 2000bp (branch [gena_web_promoters_2000](https://huggingface.co/AIRI-Institute/gena-lm-bert-large-t2t/tree/gena_web_promoters_2000))

	## Examples

	### How to load pre-trained model for Masked Language Modeling
	```python
	from transformers import AutoTokenizer, AutoModel

	tokenizer = AutoTokenizer.from_pretrained('AIRI-Institute/gena-lm-bert-large-t2t')
	model = AutoModel.from_pretrained('AIRI-Institute/gena-lm-bert-large-t2t', trust_remote_code=True)

	```

	### How to load pre-trained model to fine-tune it on classification task
	Get model class from GENA-LM repository:
	```bash
	git clone https://github.com/AIRI-Institute/GENA_LM.git
	```

	```python
	from GENA_LM.src.gena_lm.modeling_bert import BertForSequenceClassification
	from transformers import AutoTokenizer

	tokenizer = AutoTokenizer.from_pretrained('AIRI-Institute/gena-lm-bert-large-t2t')
	model = BertForSequenceClassification.from_pretrained('AIRI-Institute/gena-lm-bert-large-t2t')
	```
	or you can just download [modeling_bert.py](https://github.com/AIRI-Institute/GENA_LM/tree/main/src/gena_lm) and put it close to your code.

	OR you can get model class from HuggingFace AutoModel:
	```python
	from transformers import AutoTokenizer, AutoModel
	model = AutoModel.from_pretrained('AIRI-Institute/gena-lm-bert-large-t2t', trust_remote_code=True)
	gena_module_name = model.__class__.__module__
	print(gena_module_name)
	import importlib
	# available class names:
	# - BertModel, BertForPreTraining, BertForMaskedLM, BertForNextSentencePrediction,
	# - BertForSequenceClassification, BertForMultipleChoice, BertForTokenClassification,
	# - BertForQuestionAnswering
	# check https://huggingface.co/docs/transformers/model_doc/bert
	cls = getattr(importlib.import_module(gena_module_name), 'BertForSequenceClassification')
	print(cls)
	model = cls.from_pretrained('AIRI-Institute/gena-lm-bert-large-t2t', num_labels=2)
	```

	## Model description
	GENA-LM (`gena-lm-bert-large-t2t`) model is trained in a masked language model (MLM) fashion, following the methods proposed in the BigBird paper by masking 15% of tokens. Model config for `gena-lm-bert-large-t2t` is similar to the `bert-large-uncased`:

	- 512 Maximum sequence length
	- 24 Layers, 16 Attention heads
	- 1024 Hidden size
	- 32k Vocabulary size

	We pre-trained `gena-lm-bert-large-t2t` using the latest T2T human genome assembly (https://www.ncbi.nlm.nih.gov/assembly/GCA_009914755.3/). The data was augmented by sampling mutations from 1000-genome SNPs (gnomAD dataset). Pre-training was performed for 1,750,000 iterations with batch size 256 and sequence length was equal to 512 tokens. We modified Transformer with [Pre-Layer normalization](https://arxiv.org/abs/2002.04745).

	## Evaluation
	For evaluation results, see our paper: https://www.biorxiv.org/content/10.1101/2023.06.12.544594v1


	## Citation
	```bibtex
	@article{GENA_LM,
	author = {Veniamin Fishman and Yuri Kuratov and Maxim Petrov and Aleksei Shmelev and Denis Shepelin and Nikolay Chekanov and Olga Kardymon and Mikhail Burtsev},
	title = {GENA-LM: A Family of Open-Source Foundational Models for Long DNA Sequences},
	elocation-id = {2023.06.12.544594},
	year = {2023},
	doi = {10.1101/2023.06.12.544594},
	publisher = {Cold Spring Harbor Laboratory},
	URL = {https://www.biorxiv.org/content/early/2023/06/13/2023.06.12.544594},
	eprint = {https://www.biorxiv.org/content/early/2023/06/13/2023.06.12.544594.full.pdf},
	journal = {bioRxiv}
	}
	```