yurakuratov commited on
Commit
93c6676
1 Parent(s): 26ac5d6

fix: typos

Browse files
Files changed (1) hide show
  1. README.md +4 -4
README.md CHANGED
@@ -4,15 +4,15 @@ tags:
4
  - human_genome
5
  ---
6
 
7
- # GENA-LM (gena-lm-bigbird-base-sparse-t2t-t2t)
8
 
9
  GENA-LM is a Family of Open-Source Foundational Models for Long DNA Sequences.
10
 
11
  GENA-LM models are transformer masked language models trained on human DNA sequence.
12
 
13
- `gena-lm-bigbird-base-sparse-t2t-t2t` follows the BigBird architecture and uses sparse attention from DeepSpeed.
14
 
15
- Differences between GENA-LM (`gena-lm-bigbird-base-sparse-t2t-t2t`) and DNABERT:
16
  - BPE tokenization instead of k-mers;
17
  - input sequence size is about 36000 nucleotides (4096 BPE tokens) compared to 512 nucleotides of DNABERT;
18
  - pre-training on T2T vs. GRCh38.p13 human genome assembly.
@@ -22,7 +22,7 @@ Source code and data: https://github.com/AIRI-Institute/GENA_LM
22
  Paper: https://www.biorxiv.org/content/10.1101/2023.06.12.544594v1
23
 
24
  ## Installation
25
- `gena-lm-bigbird-base-sparse-t2t-t2t` sparse ops require DeepSpeed.
26
 
27
  ### DeepSpeed
28
  DeepSpeed installation is needed to work with SparseAttention versions of language models. DeepSpeed Sparse attention supports only GPUs with compute compatibility >= 7 (V100, T4, A100).
 
4
  - human_genome
5
  ---
6
 
7
+ # GENA-LM (gena-lm-bigbird-base-sparse-t2t)
8
 
9
  GENA-LM is a Family of Open-Source Foundational Models for Long DNA Sequences.
10
 
11
  GENA-LM models are transformer masked language models trained on human DNA sequence.
12
 
13
+ `gena-lm-bigbird-base-sparse-t2t` follows the BigBird architecture and uses sparse attention from DeepSpeed.
14
 
15
+ Differences between GENA-LM (`gena-lm-bigbird-base-sparse-t2t`) and DNABERT:
16
  - BPE tokenization instead of k-mers;
17
  - input sequence size is about 36000 nucleotides (4096 BPE tokens) compared to 512 nucleotides of DNABERT;
18
  - pre-training on T2T vs. GRCh38.p13 human genome assembly.
 
22
  Paper: https://www.biorxiv.org/content/10.1101/2023.06.12.544594v1
23
 
24
  ## Installation
25
+ `gena-lm-bigbird-base-sparse-t2t` sparse ops require DeepSpeed.
26
 
27
  ### DeepSpeed
28
  DeepSpeed installation is needed to work with SparseAttention versions of language models. DeepSpeed Sparse attention supports only GPUs with compute compatibility >= 7 (V100, T4, A100).