File size: 1,082 Bytes
9014f5a 004188b 9014f5a 004188b a14b29e |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
---
license: apache-2.0
language:
- pl
tags:
- qlora
- instruct
---
This repo contains a qlora adapter for [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1), fine-tuned on a private instructions dataset, exclusively in Polish.
The fine-tuning took 1 hour on a single RTX 4090 with the following hyperparameters:
* context length: 2048
* batch_size: 16
* learning_rate: 0.0001
* lora_r: 64
* lora_alpha: 16
* lora_modules: all
* lora_dropout: 0.0
* weight_decay: 0.1
* max_grad_norm: 0.3
* double_quant, nf4
* optimizer: paged_adamw_32bit (beta2: 0.999)
During the training of the adapter, the instruction format was maintained. Each prompt should be surrounded by `[INST]` and `[\INST]` tokens to leverage instruction fine-tuning.
This adapter allows the model to speak Polish more accurately than vanilla [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1).
<p align="center">
<img src="https://huggingface.co/Azurro/Mistral-7B-Instruct-v0.1-qlora-polish/raw/main/mistral-7b-instruct-qlora-pl.jpg">
</p> |