Update README.md
Browse files
README.md
CHANGED
|
@@ -37,9 +37,15 @@ MistralAI 7B model fine-tuned for 1 epoch on Dataricks instruction tuning datase
|
|
| 37 |
- Quantized Low-Rank Adaptation (QLoRA)
|
| 38 |
- Transformers Trainer
|
| 39 |
- DataCollatorForSeq2Seq
|
|
|
|
| 40 |
|
| 41 |
#### Preprocessing
|
| 42 |
|
| 43 |
Manually created tokenized 'labels' for the dataset.
|
|
|
|
| 44 |
|
|
|
|
| 45 |
|
|
|
|
|
|
|
|
|
|
|
|
| 37 |
- Quantized Low-Rank Adaptation (QLoRA)
|
| 38 |
- Transformers Trainer
|
| 39 |
- DataCollatorForSeq2Seq
|
| 40 |
+
- Distributed Data Parallel (DDP) across two GPUs
|
| 41 |
|
| 42 |
#### Preprocessing
|
| 43 |
|
| 44 |
Manually created tokenized 'labels' for the dataset.
|
| 45 |
+
Prompt template utilized basic template for instruction-tuning
|
| 46 |
|
| 47 |
+
### Hardware
|
| 48 |
|
| 49 |
+
Performed fine-tuning with 2 * A100 GPUs
|
| 50 |
+
- Provided by Gnewsoft during work period
|
| 51 |
+
Model and dataset are too large for free run sessions on Google Colab
|