|
--- |
|
library_name: peft |
|
license: apache-2.0 |
|
datasets: |
|
- adriantheuma/raven-data |
|
language: |
|
- en |
|
--- |
|
### Training details |
|
|
|
* Prompt tokenisation: [LlamaTokenizer](https://huggingface.co/docs/transformers/model_doc/llama2#transformers.LlamaTokenizer). |
|
* Maximum context length: 1,204 tokens |
|
* Per device train batch: 1 |
|
* Gradient accumulation: 128 steps (achieving the equivalent batch_size of 128) |
|
* Quantisation: 8-bit |
|
* Optimiser: adamw |
|
* Learning_rate: 3 × 10−4 |
|
* warmup_steps: 100 |
|
* epochs: 5 |
|
|
|
* Low Rank Adaptation (LoRA) |
|
* rank: 16 |
|
* alpha: 16 |
|
* dropout: 0.05 |
|
* target modules: q_proj, k_proj, v_proj, and o_proj |
|
|
|
This setup reduces the trainable parameters to 26,214,400 or 0.2% of the base [Llama 2 13B Chat](https://huggingface.co/docs/transformers/model_doc/llama2) model. |
|
|
|
### Training hardware |
|
|
|
This model is trained on commodity hardware equipped with a: |
|
* 13th Gen Intel(R) Core(TM) i7-13700KF CPU at 3.40 GHz |
|
* 64 GB installed RAM |
|
* NVIDIA GeForce RTX 4090 GPU with 24 GB onboard RAM. |
|
|
|
The trained model consumed 100 GPU hours during training. |
