Can't do fine-tuning on Colab
Author: yukiarimo
Pull request: no

<head></head><p class="first:mt-0 last:mb-0" dir="ltr" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="white-space: pre-wrap;">Hello everyone. I used this code before for LLaMA 2 7B. But now, it doesn't work with any model, even Phi 3!</span></p><pre spellcheck="false" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="white-space: pre-wrap;">!pip install -q accelerate bitsandbytes peft==0.4.0 transformers==4.38.2 trl==0.4.7</span><br><span style="white-space: pre-wrap;">!pip install flash_attn</span><br><br><span style="white-space: pre-wrap;">import os</span><br><span style="white-space: pre-wrap;">import torch</span><br><span style="white-space: pre-wrap;">from datasets import load_dataset</span><br><span style="white-space: pre-wrap;">from transformers import (</span><br><span style="white-space: pre-wrap;">    AutoModelForCausalLM,</span><br><span style="white-space: pre-wrap;">    AutoTokenizer,</span><br><span style="white-space: pre-wrap;">    BitsAndBytesConfig,</span><br><span style="white-space: pre-wrap;">    HfArgumentParser,</span><br><span style="white-space: pre-wrap;">    TrainingArguments,</span><br><span style="white-space: pre-wrap;">    pipeline,</span><br><span style="white-space: pre-wrap;">    logging,</span><br><span style="white-space: pre-wrap;">)</span><br><span style="white-space: pre-wrap;">from peft import LoraConfig, PeftModel</span><br><span style="white-space: pre-wrap;">from trl import SFTTrainer</span><br><br><span style="white-space: pre-wrap;"># The model that you want to train from the Hugging Face hub</span><br><span style="white-space: pre-wrap;">model_name = "cognitivecomputations/dolphin-2.9.2-qwen2-7b"</span><br><br><span style="white-space: pre-wrap;">################################################################################</span><br><span style="white-space: pre-wrap;"># QLoRA parameters</span><br><span style="white-space: pre-wrap;">################################################################################</span><br><br><span style="white-space: pre-wrap;"># LoRA attention dimension</span><br><span style="white-space: pre-wrap;">lora_r = 64</span><br><br><span style="white-space: pre-wrap;"># Alpha parameter for LoRA scaling</span><br><span style="white-space: pre-wrap;">lora_alpha = 16</span><br><br><span style="white-space: pre-wrap;"># Dropout probability for LoRA layers</span><br><span style="white-space: pre-wrap;">lora_dropout = 0.1</span><br><br><span style="white-space: pre-wrap;">################################################################################</span><br><span style="white-space: pre-wrap;"># bitsandbytes parameters</span><br><span style="white-space: pre-wrap;">################################################################################</span><br><br><span style="white-space: pre-wrap;"># Activate 4-bit precision base model loading</span><br><span style="white-space: pre-wrap;">use_4bit = True</span><br><br><span style="white-space: pre-wrap;"># Compute dtype for 4-bit base models</span><br><span style="white-space: pre-wrap;">bnb_4bit_compute_dtype = "float16"</span><br><br><span style="white-space: pre-wrap;"># Quantization type (fp4 or nf4)</span><br><span style="white-space: pre-wrap;">bnb_4bit_quant_type = "nf4"</span><br><br><span style="white-space: pre-wrap;"># Activate nested quantization for 4-bit base models (double quantization)</span><br><span style="white-space: pre-wrap;">use_nested_quant = False</span><br><br><span style="white-space: pre-wrap;">################################################################################</span><br><span style="white-space: pre-wrap;"># TrainingArguments parameters</span><br><span style="white-space: pre-wrap;">################################################################################</span><br><br><span style="white-space: pre-wrap;"># Output directory where the model predictions and checkpoints will be stored</span><br><span style="white-space: pre-wrap;">output_dir = "/content/drive/MyDrive/qtamer"</span><br><br><span style="white-space: pre-wrap;"># Number of training epochs</span><br><span style="white-space: pre-wrap;">num_train_epochs = 1</span><br><br><span style="white-space: pre-wrap;"># Enable fp16/bf16 training (set bf16 to True with an A100)</span><br><span style="white-space: pre-wrap;">fp16 = False</span><br><span style="white-space: pre-wrap;">bf16 = False</span><br><br><span style="white-space: pre-wrap;"># Batch size per GPU for training</span><br><span style="white-space: pre-wrap;">per_device_train_batch_size = 2</span><br><br><span style="white-space: pre-wrap;"># Batch size per GPU for evaluation</span><br><span style="white-space: pre-wrap;">per_device_eval_batch_size = 2</span><br><br><span style="white-space: pre-wrap;"># Number of update steps to accumulate the gradients for</span><br><span style="white-space: pre-wrap;">gradient_accumulation_steps = 2</span><br><br><span style="white-space: pre-wrap;"># Enable gradient checkpointing</span><br><span style="white-space: pre-wrap;">gradient_checkpointing = True</span><br><br><span style="white-space: pre-wrap;"># Maximum gradient normal (gradient clipping)</span><br><span style="white-space: pre-wrap;">max_grad_norm = 0.3</span><br><br><span style="white-space: pre-wrap;"># Initial learning rate (AdamW optimizer)</span><br><span style="white-space: pre-wrap;">learning_rate = 3e-4</span><br><br><span style="white-space: pre-wrap;"># Weight decay to apply to all layers except bias/LayerNorm weights</span><br><span style="white-space: pre-wrap;">weight_decay = 0.001</span><br><br><span style="white-space: pre-wrap;"># Optimizer to use</span><br><span style="white-space: pre-wrap;">optim = "paged_adamw_32bit"</span><br><br><span style="white-space: pre-wrap;"># Learning rate schedule</span><br><span style="white-space: pre-wrap;">lr_scheduler_type = "cosine"</span><br><br><span style="white-space: pre-wrap;"># Number of training steps (overrides num_train_epochs)</span><br><span style="white-space: pre-wrap;">max_steps = 4000</span><br><br><span style="white-space: pre-wrap;"># Ratio of steps for a linear warmup (from 0 to learning rate)</span><br><span style="white-space: pre-wrap;">warmup_ratio = 0.02</span><br><br><span style="white-space: pre-wrap;"># Group sequences into batches with same length</span><br><span style="white-space: pre-wrap;"># Saves memory and speeds up training considerably</span><br><span style="white-space: pre-wrap;">group_by_length = True</span><br><br><span style="white-space: pre-wrap;"># Save checkpoint every X updates steps</span><br><span style="white-space: pre-wrap;">save_steps = 10</span><br><br><span style="white-space: pre-wrap;"># Log every X updates steps</span><br><span style="white-space: pre-wrap;">logging_steps = 1</span><br><br><span style="white-space: pre-wrap;">################################################################################</span><br><span style="white-space: pre-wrap;"># SFT parameters</span><br><span style="white-space: pre-wrap;">################################################################################</span><br><br><span style="white-space: pre-wrap;"># Maximum sequence length to use</span><br><span style="white-space: pre-wrap;">max_seq_length = 2048 #2048</span><br><br><span style="white-space: pre-wrap;"># Pack multiple short examples in the same input sequence to increase efficiency</span><br><span style="white-space: pre-wrap;">packing = False</span><br><br><span style="white-space: pre-wrap;"># Load the entire model on the GPU 0</span><br><span style="white-space: pre-wrap;">device_map = {"": 0}</span><br><br><span style="white-space: pre-wrap;">from datasets import load_dataset</span><br><br><span style="white-space: pre-wrap;">def formatting_prompts_func(examples):</span><br><span style="white-space: pre-wrap;">    texts = examples["note"]</span><br><span style="white-space: pre-wrap;">    return {"note": texts}</span><br><br><span style="white-space: pre-wrap;">dataset = load_dataset("json", data_files="/content/drive/MyDrive/YunaAI/yuna-ai-v3-text-datasets/2048.jsonl", split="train")</span><br><span style="white-space: pre-wrap;">#dataset = dataset.map(formatting_prompts_func, batched=True)</span><br><br><span style="white-space: pre-wrap;"># Load dataset (you can process it here)</span><br><br><span style="white-space: pre-wrap;"># Load tokenizer and model with QLoRA configuration</span><br><span style="white-space: pre-wrap;">compute_dtype = getattr(torch, bnb_4bit_compute_dtype)</span><br><br><span style="white-space: pre-wrap;">bnb_config = BitsAndBytesConfig(</span><br><span style="white-space: pre-wrap;">    load_in_4bit=use_4bit,</span><br><span style="white-space: pre-wrap;">    bnb_4bit_quant_type=bnb_4bit_quant_type,</span><br><span style="white-space: pre-wrap;">    bnb_4bit_compute_dtype=compute_dtype,</span><br><span style="white-space: pre-wrap;">    bnb_4bit_use_double_quant=use_nested_quant,</span><br><span style="white-space: pre-wrap;">)</span><br><br><span style="white-space: pre-wrap;"># Check GPU compatibility with bfloat16</span><br><span style="white-space: pre-wrap;">if compute_dtype == torch.float16 and use_4bit:</span><br><span style="white-space: pre-wrap;">    major, _ = torch.cuda.get_device_capability()</span><br><span style="white-space: pre-wrap;">    if major &gt;= 8:</span><br><span style="white-space: pre-wrap;">        print("=" * 80)</span><br><span style="white-space: pre-wrap;">        print("Your GPU supports bfloat16: accelerate training with bf16=True")</span><br><span style="white-space: pre-wrap;">        print("=" * 80)</span><br><br><span style="white-space: pre-wrap;"># Load base model</span><br><span style="white-space: pre-wrap;">model = AutoModelForCausalLM.from_pretrained(</span><br><span style="white-space: pre-wrap;">    model_name,</span><br><span style="white-space: pre-wrap;">    quantization_config=bnb_config,</span><br><span style="white-space: pre-wrap;">    device_map=device_map</span><br><span style="white-space: pre-wrap;">)</span><br><span style="white-space: pre-wrap;">model.config.use_cache = False</span><br><span style="white-space: pre-wrap;">model.config.pretraining_tp = 1</span><br><br><span style="white-space: pre-wrap;"># Load LLaMA tokenizer</span><br><span style="white-space: pre-wrap;">tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)</span><br><span style="white-space: pre-wrap;">tokenizer.pad_token = tokenizer.eos_token</span><br><span style="white-space: pre-wrap;">tokenizer.padding_side = "right" # Fix weird overflow issue with fp16 training</span><br><br><span style="white-space: pre-wrap;"># Load LoRA configuration</span><br><span style="white-space: pre-wrap;">peft_config = LoraConfig(</span><br><span style="white-space: pre-wrap;">    lora_alpha=lora_alpha,</span><br><span style="white-space: pre-wrap;">    lora_dropout=lora_dropout,</span><br><span style="white-space: pre-wrap;">    r=lora_r,</span><br><span style="white-space: pre-wrap;">    target_modules = ["q_proj", "k_proj", "v_proj", "o_proj",</span><br><span style="white-space: pre-wrap;">                      "gate_proj", "up_proj", "down_proj",],</span><br><span style="white-space: pre-wrap;">    bias="none",</span><br><span style="white-space: pre-wrap;">    task_type="CAUSAL_LM",</span><br><span style="white-space: pre-wrap;">)</span><br><br><span style="white-space: pre-wrap;"># Set training parameters</span><br><span style="white-space: pre-wrap;">training_arguments = TrainingArguments(</span><br><span style="white-space: pre-wrap;">    output_dir=output_dir,</span><br><span style="white-space: pre-wrap;">    num_train_epochs=num_train_epochs,</span><br><span style="white-space: pre-wrap;">    per_device_train_batch_size=per_device_train_batch_size,</span><br><span style="white-space: pre-wrap;">    gradient_accumulation_steps=gradient_accumulation_steps,</span><br><span style="white-space: pre-wrap;">    optim=optim,</span><br><span style="white-space: pre-wrap;">    save_steps=save_steps,</span><br><span style="white-space: pre-wrap;">    logging_steps=logging_steps,</span><br><span style="white-space: pre-wrap;">    learning_rate=learning_rate,</span><br><span style="white-space: pre-wrap;">    weight_decay=weight_decay,</span><br><span style="white-space: pre-wrap;">    fp16=fp16,</span><br><span style="white-space: pre-wrap;">    bf16=bf16,</span><br><span style="white-space: pre-wrap;">    max_grad_norm=max_grad_norm,</span><br><span style="white-space: pre-wrap;">    max_steps=max_steps,</span><br><span style="white-space: pre-wrap;">    warmup_ratio=warmup_ratio,</span><br><span style="white-space: pre-wrap;">    group_by_length=group_by_length,</span><br><span style="white-space: pre-wrap;">    lr_scheduler_type=lr_scheduler_type,</span><br><span style="white-space: pre-wrap;">    report_to="tensorboard"</span><br><span style="white-space: pre-wrap;">)</span><br><br><span style="white-space: pre-wrap;"># Set supervised fine-tuning parameters</span><br><span style="white-space: pre-wrap;">trainer = SFTTrainer(</span><br><span style="white-space: pre-wrap;">    model=model,</span><br><span style="white-space: pre-wrap;">    train_dataset=dataset,</span><br><span style="white-space: pre-wrap;">    peft_config=peft_config,</span><br><span style="white-space: pre-wrap;">    dataset_text_field="note",</span><br><span style="white-space: pre-wrap;">    max_seq_length=max_seq_length,</span><br><span style="white-space: pre-wrap;">    tokenizer=tokenizer,</span><br><span style="white-space: pre-wrap;">    args=training_arguments,</span><br><span style="white-space: pre-wrap;">    packing=packing,</span><br><span style="white-space: pre-wrap;">)</span><br><br><span style="white-space: pre-wrap;"># Train model</span><br><span style="white-space: pre-wrap;">trainer.train()</span><br><br><span style="white-space: pre-wrap;"># Save trained model</span><br><span style="white-space: pre-wrap;">trainer.model.save_pretrained(new_model)</span></pre><p class="first:mt-0 last:mb-0" dir="ltr" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="white-space: pre-wrap;">And here's my logs:</span></p><pre spellcheck="false" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="white-space: pre-wrap;">Your GPU supports bfloat16: accelerate training with bf16=True</span><br><span style="white-space: pre-wrap;">================================================================================</span><br><br><span style="white-space: pre-wrap;">/usr/local/lib/python3.10/dist-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.</span><br><span style="white-space: pre-wrap;">  warnings.warn(</span><br><span style="white-space: pre-wrap;">Loading checkpoint shards: 100% 4/4 [00:10&lt;00:00,  2.24s/it]</span><br><br><span style="white-space: pre-wrap;">Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.</span><br><span style="white-space: pre-wrap;">/usr/local/lib/python3.10/dist-packages/peft/utils/other.py:102: FutureWarning: prepare_model_for_int8_training is deprecated and will be removed in a future version. Use prepare_model_for_kbit_training instead.</span><br><span style="white-space: pre-wrap;">  warnings.warn(</span><br><span style="white-space: pre-wrap;">/usr/local/lib/python3.10/dist-packages/accelerate/accelerator.py:444: FutureWarning: Passing the following arguments to `Accelerator` is deprecated and will be removed in version 1.0 of Accelerate: dict_keys(['dispatch_batches', 'split_batches', 'even_batches', 'use_seedable_sampler']). Please pass an `accelerate.DataLoaderConfiguration` instead: </span><br><span style="white-space: pre-wrap;">dataloader_config = DataLoaderConfiguration(dispatch_batches=None, split_batches=False, even_batches=True, use_seedable_sampler=True)</span><br><span style="white-space: pre-wrap;">  warnings.warn(</span><br><span style="white-space: pre-wrap;"> [ 4/4000 01:30 &lt; 50:08:35, 0.02 it/s, Epoch 0.03/44]</span></pre>
Step | Training Loss
-- | --
1 | 2.420900
2 | 2.398200

<p class="first:mt-0 last:mb-0" dir="ltr" style="font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none; caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0);"><span style="white-space: pre-wrap;">It shows 50 hours instead of 30 minutes as before. My GPUs is L4. Any idea what's going wrong?</span></p>

Comments:
Hi @yukiarimo, thanks for the issue!
Is there any chance that you ran out of GPU credits on colab and you are training on CPU instead of GPU?

No, I have 100 credits, and 19 GB of VRAM is used. Is this `device_map = {": 0}` working fine, or should I modify it?

cc @SunMarc 

Thanks for reporting @yukiarimo ! 

>  Is this device_map = {": 0} working fine, or should I modify it?

This should be fine. Which version of transformer, peft and accelerate were you using previously ? 

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.