| --- | |
| ## Model Details | |
| I finetuned PygmalionAI/pygmalion-6b with QLora for 24 hours on 250k samples. Collected from SODA and Teacher GPT dataset. My first attempt on making LLM model as an entry to Chai competition. | |
| ### Model Description | |
| - **Model type:** Chatbot | |
| - **Finetuned from model :** PygmalionAI/pygmalion-6b | |
| ### Model Sources | |
| Pygmalion-6b: https://huggingface.co/PygmalionAI/pygmalion-6b | |
| ## Training Details | |
| ### Training Data | |
| For the training data I use 20% of SODA dadtaset mixed with TeacherGPT roleplay dataset. | |
| ### Training Procedure | |
| The model was trained for 24 hours on RTX4090. | |
| #### Training Hyperparameters | |
| - Training param | |
| >batch_size = 128, | |
| >micro_batch_size = 4, | |
| >num_epochs = 1, | |
| >learning_rate = 3e-4, | |
| >cutoff_len = 512, | |
| >val_set_size = 0 | |
| - finetune method | |
| >finetune_method = "qlora" | |
| - prefix tuning hyperparams | |
| >num_virtual_tokens = 32 | |
| - lora hyperparams | |
| >lora_r = 16, | |
| >lora_alpha = 16, | |
| >lora_dropout = 0.05, | |
| >lora_target_modules = "q_proj k_proj v_proj" | |
| - llm hyperparams | |
| >bf16 = False, | |
| >load_in_8bit = False, | |
| >group_by_length = False , | |
| >resume_from_checkpoint = None | |
| ### Results | |
| Me: Hi Nathan, how are you doing today | |
| Nathan: I'm fine... | |
| Me: Then tell me about your day. | |
| Nathan: | |
| -- | |
| It was good. We had a lot of fun in school and then we went to the park afterwards. |