Update README.md

README.md

@@ -1,3 +1,26 @@
 ---
 license: cc-by-nc-4.0
+language:
+- pl
+tags:
+- llama
+- qlora
 ---
+
+This repo contains a qlora adapter for [Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b-hf), trained on 1B tokens, only in Polish.
+
+The training took 20 days on a single RTX 4090 with the following hyperparameters:
+
+* context length: 4096
+* batch_size: 128
+* learning_rate: 0.0002, cosine with warmup
+* lora_r: 64
+* lora_alpha: 16
+* lora_modules: all
+* lora_dropout: 0.0
+* weight_decay: 0.1
+* max_grad_norm: 0.3
+* double_quant, nf4
+* optimizer: paged_adamw_32bit
+
+This adapter allows the model to speak Polish more accurately than vanilla [Llama-2-7b](https://huggingface.co/meta-llama/Llama-2-7b-hf).