training t5-11b on 32GB GPU V100
Author: Arij-Aladel
Pull request: no


Comments:
@stas00 Just quick question please: could we use the same steps to finetune t5-11b or ul2 on single gpu with 32GB memory V100?  Or I need 40GB memory? 
if yes, what is maximim input length which can I use?
Today I have tried to finetune t5-11b using this yaml config

`
compute_environment: LOCAL_MACHINE
deepspeed_config:
  offload_optimizer_device: cpu
  zero_stage: 2
distributed_type: DEEPSPEE
downcast_bf16: 'no'
fsdp_config: {}
machine_rank: 0
main_training_function: main
main_process_port: 20683
mixed_precision: 'no'
num_machines: 1
num_processes: 1
rdzv_backend: static
same_network: true
use_cpu: false`

for model I have loaded it as folloing:

`
        # Load model
        peft_config = LoraConfig(
            task_type=TaskType.SEQ_2_SEQ_LM,
            inference_mode=False,
            r=8,
            lora_alpha=32,
            target_modules=["q", "v"],
            lora_dropout=0.1,
        )

          model = AutoModelForSeq2SeqLM.from_pretrained(
              args.model_name_or_path, # t5-11b
              load_in_8bit=True,
              device_map="auto",
            )
`

this is my run script

`export   CUDA_LAUNCH_BLOCKING=0 CUDA_VISIBLE_DEVICES=5 WANDB_CONSOLE=off TORCH_DISTRIBUTED_DEBUG=INFO
accelerate launch --config_file='./accelerate.yaml' train_seqtoseq_PEFT.py  --seed=42 --preprocessing_num_workers=1 --weight_decay='0.001' --output_dir="PEFT_t5_11b_kv" --per_device_train_batch_size=1 --per_device_eval_batch_size=1    --num_train_epochs=10  --model_name_or_path='t5-11b' --num_beams=5 --with_tracking --report_to='wandb' --checkpointing_steps='epoch' --dataset_name="databricks-dolly-15k" --dataset_config "3.0.0"   --gradient_accumulation_steps=1 --tokenizer_name='t5-11b' --max_target_length=1024`

I got this error:

trainable params: 5,898,240 || all params: 2,857,496,576 || trainable%: 0.20641284575943442
trainable params: 5,898,240 || all params: 2,857,496,576 || trainable%: 0.20641284575943442
09/24/2023 11:48:35 - INFO - root - model.print_trainable_parameters():
ns key of the tokenizer
dolly before add text columns:   ['instruction', 'context', 'response', 'category']
dolly after add text columns:   ['input', 'target']
raw_datasets[0]:   {'input': "### Instruction: \nWhen did Virgin Australia start operating?\n### Context: \nVirgin Australia, the trading name of Virgin Australia Airlines Pty Ltd, is an Australian-base
Running tokenizer on dataset: 100%|█████████████████████| 1502/1502 [00:01<00:00, 1409.67 examples/s]
    validation: Dataset({
09/24/2023 11:48:48 - INFO - root - Val dataset length :1501
19, 3, 9, 11820, 40, 11417, 58, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1], 'labels': [1255
5146, 13, 331, 6, 11, 79, 54, 92, 560, 650, 42, 1561, 3, 31761, 2865, 147, 97, 5, 12558, 13154, 661, 
s to 8 from 1.
Using /home/arij/.cache/torch_extensions/py39_cu117 as PyTorch extensions root...
Emitting ninja build file /home/arij/.cache/torch_extensions/py39_cu117/cpu_adam/build.ninja...
er for key:store_based_barrier_key:2 with 1 nodes.
  File "/mnt/ssd/arij/NeurIPS/NeurIPSS/lib/python3.9/site-packages/accelerate/accelerator.py", line 1
 in log_wrapper
    return func(*args, **kwargs)
  File "/mnt/ssd/arij/NeurIPS/NeurIPSS/lib/python3.9/site-packages/deepspeed/comm/comm.py", line 224, in broadcast
    return cdb.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)
  File "/mnt/ssd/arij/NeurIPS/NeurIPSS/lib/python3.9/site-packages/deepspeed/comm/torch.py", line 192, in broadcast
    return torch.distributed.broadcast(tensor=tensor, src=src, group=group, async_op=async_op)
  File "/mnt/ssd/arij/NeurIPS/NeurIPSS/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1451, in wrapper
    return func(*args, **kwargs)
  File "/mnt/ssd/arij/NeurIPS/NeurIPSS/lib/python3.9/site-packages/torch/distributed/distributed_c10d.py", line 1570, in broadcast
    work = group.broadcast([tensor], opts)
RuntimeError: Tensors must be contiguous

What had I missed?



tagging @pacman100 who is the current maintainer of the Accelerate/Deepspeed integration.

@pacman100 I have tried ith t5-3b loss is nan after first epoch and accuracy metric 0   :/

@stas00 who can help?

Hello @Arij-Aladel, DeepSpeed and BitsandBytes aren't compatible with each other. Could you please try to use only BitsandBytes + PEFT for training 11B on a single 32GB V100 GPU.