---
## Model Details

I finetuned PygmalionAI/pygmalion-6b with QLora for 24 hours on 250k samples. Collected from SODA and Teacher GPT dataset. My first attempt on making LLM model as an entry to Chai competition.

### Model Description

- **Model type:** Chatbot
- **Finetuned from model :** PygmalionAI/pygmalion-6b

### Model Sources

Pygmalion-6b: https://huggingface.co/PygmalionAI/pygmalion-6b

## Training Details

### Training Data

For the training data I use 20% of SODA dadtaset mixed with TeacherGPT roleplay dataset.

### Training Procedure 

The model was trained for 24 hours on RTX4090.

#### Training Hyperparameters

- Training param
>batch_size = 128,          
>micro_batch_size = 4,          
>num_epochs = 1,          
>learning_rate = 3e-4,          
>cutoff_len = 512,          
>val_set_size = 0

- finetune method
>finetune_method = "qlora"

- prefix tuning hyperparams
>num_virtual_tokens = 32

- lora hyperparams
>lora_r = 16,          
>lora_alpha = 16,          
>lora_dropout = 0.05,          
>lora_target_modules = "q_proj k_proj v_proj"

- llm hyperparams
>bf16 = False,          
>load_in_8bit = False,          
>group_by_length = False ,         
>resume_from_checkpoint = None

### Results

Me: Hi Nathan, how are you doing today       
Nathan: I'm fine...       
Me: Then tell me about your day.      
Nathan:        
--        
It was good. We had a lot of fun in school and then we went to the park afterwards.