Commit
•
f422c79
1
Parent(s):
4fda816
Update README.md
Browse files
README.md
CHANGED
|
@@ -58,13 +58,13 @@ reference-data-model:
|
|
| 58 |
|
| 59 |
|
| 60 |
```py
|
| 61 |
-
|
| 62 |
-
# Activated training with
|
| 63 |
-
#- https://huggingface.co/blog/tomaarsen/attention-sinks
|
| 64 |
-
#- https://github.com/tomaarsen/attention_sinks
|
| 65 |
-
#- https://arxiv.org/abs/2309.17453
|
| 66 |
-
|
| 67 |
pip install attention_sinks
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 68 |
```
|
| 69 |
|
| 70 |
|
|
|
|
| 58 |
|
| 59 |
|
| 60 |
```py
|
| 61 |
+
# attention-sinks
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 62 |
pip install attention_sinks
|
| 63 |
+
|
| 64 |
+
# flash-attn
|
| 65 |
+
!export CUDA_HOME=/usr/local/cuda-11.8
|
| 66 |
+
!MAX_JOBS=4 pip install flash-attn --no-build-isolation -qqq
|
| 67 |
+
!pip install git+"https://github.com/HazyResearch/flash-attention.git#subdirectory=csrc/rotary" -qqq
|
| 68 |
```
|
| 69 |
|
| 70 |
|