Create README.md

README.md

@@ -0,0 +1,61 @@
+---
+## Model Details
+
+I trained PygmalionAI/pygmalion-6b with QLora for 24 hours on 250k samples. Collected from SODA and Teacher GPT dataset.
+
+### Model Description
+
+- **Developed by:** Me
+- **Model type:** Chatbot
+- **Finetuned from model [optional]:** PygmalionAI/pygmalion-6b
+
+### Model Sources
+
+Pygmalion-6b: https://huggingface.co/PygmalionAI/pygmalion-6b
+
+## Training Details
+
+### Training Data
+
+For the training data I use 20% of SODA dadtaset mixed with TeacherGPT roleplay dataset.
+
+### Training Procedure 
+
+The model was trained for 24 hours on RTX4090.
+
+#### Training Hyperparameters
+
+- Training param
+>batch_size = 128,          
+>micro_batch_size = 4,          
+>num_epochs = 1,          
+>learning_rate = 3e-4,          
+>cutoff_len = 512,          
+>val_set_size = 0
+
+- finetune method
+>finetune_method = "qlora"
+
+- prefix tuning hyperparams
+>num_virtual_tokens = 32
+
+- lora hyperparams
+>lora_r = 16,          
+>lora_alpha = 16,          
+>lora_dropout = 0.05,          
+>lora_target_modules = "q_proj k_proj v_proj"
+
+- llm hyperparams
+>bf16 = False,          
+>load_in_8bit = False,          
+>group_by_length = False ,         
+>resume_from_checkpoint = None
+
+### Results
+
+Me: Hi Nathan, how are you doing today       
+Nathan: I'm fine...       
+Me: Then tell me about your day.      
+Nathan:        
+--        
+It was good. We had a lot of fun in school and then we went to the park afterwards.