yurakuratov
commited on
Commit
•
93c6676
1
Parent(s):
26ac5d6
fix: typos
Browse files
README.md
CHANGED
@@ -4,15 +4,15 @@ tags:
|
|
4 |
- human_genome
|
5 |
---
|
6 |
|
7 |
-
# GENA-LM (gena-lm-bigbird-base-sparse-t2t
|
8 |
|
9 |
GENA-LM is a Family of Open-Source Foundational Models for Long DNA Sequences.
|
10 |
|
11 |
GENA-LM models are transformer masked language models trained on human DNA sequence.
|
12 |
|
13 |
-
`gena-lm-bigbird-base-sparse-t2t
|
14 |
|
15 |
-
Differences between GENA-LM (`gena-lm-bigbird-base-sparse-t2t
|
16 |
- BPE tokenization instead of k-mers;
|
17 |
- input sequence size is about 36000 nucleotides (4096 BPE tokens) compared to 512 nucleotides of DNABERT;
|
18 |
- pre-training on T2T vs. GRCh38.p13 human genome assembly.
|
@@ -22,7 +22,7 @@ Source code and data: https://github.com/AIRI-Institute/GENA_LM
|
|
22 |
Paper: https://www.biorxiv.org/content/10.1101/2023.06.12.544594v1
|
23 |
|
24 |
## Installation
|
25 |
-
`gena-lm-bigbird-base-sparse-t2t
|
26 |
|
27 |
### DeepSpeed
|
28 |
DeepSpeed installation is needed to work with SparseAttention versions of language models. DeepSpeed Sparse attention supports only GPUs with compute compatibility >= 7 (V100, T4, A100).
|
|
|
4 |
- human_genome
|
5 |
---
|
6 |
|
7 |
+
# GENA-LM (gena-lm-bigbird-base-sparse-t2t)
|
8 |
|
9 |
GENA-LM is a Family of Open-Source Foundational Models for Long DNA Sequences.
|
10 |
|
11 |
GENA-LM models are transformer masked language models trained on human DNA sequence.
|
12 |
|
13 |
+
`gena-lm-bigbird-base-sparse-t2t` follows the BigBird architecture and uses sparse attention from DeepSpeed.
|
14 |
|
15 |
+
Differences between GENA-LM (`gena-lm-bigbird-base-sparse-t2t`) and DNABERT:
|
16 |
- BPE tokenization instead of k-mers;
|
17 |
- input sequence size is about 36000 nucleotides (4096 BPE tokens) compared to 512 nucleotides of DNABERT;
|
18 |
- pre-training on T2T vs. GRCh38.p13 human genome assembly.
|
|
|
22 |
Paper: https://www.biorxiv.org/content/10.1101/2023.06.12.544594v1
|
23 |
|
24 |
## Installation
|
25 |
+
`gena-lm-bigbird-base-sparse-t2t` sparse ops require DeepSpeed.
|
26 |
|
27 |
### DeepSpeed
|
28 |
DeepSpeed installation is needed to work with SparseAttention versions of language models. DeepSpeed Sparse attention supports only GPUs with compute compatibility >= 7 (V100, T4, A100).
|