File size: 1,452 Bytes
8497ff4 f9cdd37 8497ff4 f9cdd37 8497ff4 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 |
---
## Model Details
I finetuned PygmalionAI/pygmalion-6b with QLora for 24 hours on 250k samples. Collected from SODA and Teacher GPT dataset. My first attempt on making LLM model as an entry to Chai competition.
### Model Description
- **Model type:** Chatbot
- **Finetuned from model :** PygmalionAI/pygmalion-6b
### Model Sources
Pygmalion-6b: https://huggingface.co/PygmalionAI/pygmalion-6b
## Training Details
### Training Data
For the training data I use 20% of SODA dadtaset mixed with TeacherGPT roleplay dataset.
### Training Procedure
The model was trained for 24 hours on RTX4090.
#### Training Hyperparameters
- Training param
>batch_size = 128,
>micro_batch_size = 4,
>num_epochs = 1,
>learning_rate = 3e-4,
>cutoff_len = 512,
>val_set_size = 0
- finetune method
>finetune_method = "qlora"
- prefix tuning hyperparams
>num_virtual_tokens = 32
- lora hyperparams
>lora_r = 16,
>lora_alpha = 16,
>lora_dropout = 0.05,
>lora_target_modules = "q_proj k_proj v_proj"
- llm hyperparams
>bf16 = False,
>load_in_8bit = False,
>group_by_length = False ,
>resume_from_checkpoint = None
### Results
Me: Hi Nathan, how are you doing today
Nathan: I'm fine...
Me: Then tell me about your day.
Nathan:
--
It was good. We had a lot of fun in school and then we went to the park afterwards. |