alkahestry/pygmalion-6b-chaicomp

---

Model Details

I finetuned PygmalionAI/pygmalion-6b with QLora for 24 hours on 250k samples. Collected from SODA and Teacher GPT dataset. My first attempt on making LLM model as an entry to Chai competition.

Model Description

• Model type: Chatbot
• Finetuned from model : PygmalionAI/pygmalion-6b

Model Sources

Pygmalion-6b: https://huggingface.co/PygmalionAI/pygmalion-6b

Training Details

Training Data

For the training data I use 20% of SODA dadtaset mixed with TeacherGPT roleplay dataset.

Training Procedure

The model was trained for 24 hours on RTX4090.

Training Hyperparameters

• Training param

  batch_size = 128,
  micro_batch_size = 4,
  num_epochs = 1,
  learning_rate = 3e-4,
  cutoff_len = 512,
  val_set_size = 0

• finetune method

  finetune_method = "qlora"

• prefix tuning hyperparams

  num_virtual_tokens = 32

• lora hyperparams

  lora_r = 16,
  lora_alpha = 16,
  lora_dropout = 0.05,
  lora_target_modules = "q_proj k_proj v_proj"

• llm hyperparams

  bf16 = False,
  load_in_8bit = False,
  group_by_length = False ,
  resume_from_checkpoint = None

Results

Me: Hi Nathan, how are you doing today
Nathan: I'm fine...
Me: Then tell me about your day.
Nathan:

It was good. We had a lot of fun in school and then we went to the park afterwards.