Base model
bigscience/bloomz-560m
Training procedure
According to edX Databricks llm102 course
PromptTuningConfig
- task_type=TaskType.CAUSAL_LM,
- prompt_tuning_init=PromptTuningInit.RANDOM,
- num_virtual_tokens=4,
TrainingArguments
- learning_rate= 3e-2, # Higher learning rate than full fine-tuning
- num_train_epochs=5 # Number of passes to go through the entire fine-tuning dataset
Framework versions
- PEFT 0.4.0
Training output
TrainOutput(global_step=35, training_loss=3.386413792201451, metrics={'train_runtime': 617.1546, 'train_samples_per_second': 0.405, 'train_steps_per_second': 0.057, 'total_flos': 58327152033792.0, 'train_loss': 3.386413792201451, 'epoch': 5.0})
- Downloads last month
- 2