Are there memory leaks when using DeepSpeed on training T5?
Author: avionkmh
Pull request: no
Labels: name: DeepSpeed

We’ve been pretraining a T5-small model from scratch with using DeepSpeed v0.3.10.
We’ve found that cpu memory was increasing over time. (We’ve trained about 150 hours)
Are there memory leaks when using DeepSpeed on training T5?


Comments:
Hi, we'll need a bit more information to understand what's going on here. What command did you use? Did you use one of our scripts? What data are you using? What version of Transformers? The more information, the more we'll be able to understand and help you.

Pinging @stas00 

Thank you for reporting this, @avionkmh 

I've been only doing short functionality tests so far, so can't really tell.

The only general RAM leak I found so far is when `deepspeed.initialize` is called more than once and the fix is here:
https://github.com/microsoft/DeepSpeed/issues/879
I suppose this is not your case.

As @LysandreJik recommended we need a lot more details to reproduce the problem and then we or the Deepspeed team if it's in their land can fix it.



@LysandreJik @stas00 

Thank you for your interests.  

Here are the more detailed situations.

- Command
```
python -u -m deepspeed.launcher.launch \
    --world_info=eyJsb2NhbGhvc3QiOiBbMywgNF19 --master_addr=127.0.0.1 --master_port=29503 \
    examples/seq2seq/finetune_trainer.py \
    --overwrite_output_dir \
    --output_dir ./output \
    --data_dir ./input \
    --model_name_or_path ./t5-small-empty \
    --per_device_train_batch_size 16 --gradient_accumulation_steps 2 \
    --logging_steps 500 \
    --save_steps 10000 \
    --warmup_steps 10 \
    --num_train_epochs 8 \
    --deepspeed run_ds_config-cpu_offload=X.json \
    --do_train

```


- the script we used
examples/seq2seq/finetune_trainer.py 
<-- We modified the script(finetune_trainer.py) for T5 pretraining


- Version of Transformers
v4.3.2 


- DeepSpeed version: v0.3.10


- "run_ds_config-cpu_offload=X.json" file
```
{
    "fp16": {
        "enabled": true,
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "hysteresis": 2,
        "min_loss_scale": 1
    },

   "zero_optimization": {
       "stage": 2,
       "allgather_partitions": true,
       "allgather_bucket_size": 2e8,
       "overlap_comm": true,
       "reduce_scatter": true,
       "reduce_bucket_size": 2e8,
       "contiguous_gradients": true,
       "cpu_offload": false
   },

   "zero_allow_untested_optimizer": true,

   "optimizer": {
     "type": "AdamW",
     "params": {
       "lr": 5e-5,
       "betas": [
         0.9,
         0.999
       ],
       "eps": 1e-8,
       "weight_decay": 0.0
     }
   },

   "scheduler": {
     "type": "WarmupLR",
     "params": {
       "warmup_min_lr": 0,
       "warmup_max_lr": 5e-5,
       "warmup_num_steps": 10
     }
   },

    "steps_per_print": 2000,
    "wall_clock_breakdown": false
}
```


- when using "torch.distributed.launch"
We've found there was no memory leak until the end of pretraining.
The followings are the script for using "torch.distributed.launch"
```
export NODE_RANK=0
export N_NODES=1
export N_GPU_NODE=2
export WORLD_SIZE=2
export MASTER_ADDR="129.254.164.234"
export MASTER_PORT=1233

python -m torch.distributed.launch \
    --nproc_per_node=$N_GPU_NODE \
    --nnodes=$N_NODES \
    --node_rank $NODE_RANK \
    --master_addr $MASTER_ADDR \
    --master_port $MASTER_PORT \
    examples/seq2seq/finetune_trainer.py \
    --model_name_or_path t5-small-empty \
    --output_dir ./output \
    --data_dir ./input   \
    --do_train \
    --save_steps 10000 \
    --per_device_train_batch_size 16 \
    --gradient_accumulation_steps 2 \
    --num_train_epochs 8 \
    --overwrite_output_dir
```



Unfortunately, there is nothing we can do w/o you providing us a way to reproduce the problem in a simple to setup and quick to run script.

> We modified this scripts for T5 pretraining

How could we possibly know what that means?

> We've found there was no memory leak until the end of pretraining.

I'd love to help, but I have no idea what to do with this information. 

Please try to put yourself in the shoes of someone who isn't sitting in front of your computer seeing your software and what you're doing and what are the problems that you're seeing.

If we continue please first sync your code base to the latest training scripts and `transformers` since many issues have been fixed in deepspeed integration since the version you're using.

p.s. also when pasting code/config files please use code formatting as what you shared above is very difficult to read. Thank you.

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.