unable to use multiple GPUs with HF integration of DeepSpeed on Jupyter notebooks
Author: saichandrapandraju
Pull request: no
Labels: name: DeepSpeed

Hi ,

I'm using HF integration of DeepSpeed in my Jupyter Notebook by setting following env variables as suggested [here](https://huggingface.co/transformers/main_classes/trainer.html#deployment-in-notebooks)

```
os.environ['MASTER_ADDR'] = 'localhost'
os.environ['MASTER_PORT'] = '9889'
os.environ['RANK'] = "0"
os.environ['LOCAL_RANK'] = "0"
os.environ['WORLD_SIZE'] = "1"
os.environ['NCCL_SOCKET_IFNAME'] = 'lo'   ##because of my kubeflow setup
```
with this setup I'm unable to utilize both GPU's that I have and here's the log info before it starts training- 
```
[2021-03-31 09:49:02,510] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.13+7fcc891, git-hash=7fcc891, git-branch=master
[2021-03-31 09:49:02,540] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 1, parameter_parallel_size: 1
[2021-03-31 09:49:05,758] [INFO] [engine.py:608:_configure_optimizer] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2021-03-31 09:49:05,760] [INFO] [engine.py:612:_configure_optimizer] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2021-03-31 09:49:05,761] [INFO] [logging.py:60:log_dist] [Rank 0] Creating fp16 ZeRO stage 2 optimizer
[2021-03-31 09:49:05,764] [INFO] [stage2.py:130:__init__] Reduce bucket size 150000000.0
[2021-03-31 09:49:05,765] [INFO] [stage2.py:131:__init__] Allgather bucket size 150000000.0
[2021-03-31 09:49:05,766] [INFO] [stage2.py:132:__init__] CPU Offload: True
[2021-03-31 09:49:12,585] [INFO] [stage2.py:399:__init__] optimizer state initialized
[2021-03-31 09:49:12,588] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
[2021-03-31 09:49:12,589] [INFO] [engine.py:445:_configure_lr_scheduler] DeepSpeed using configured LR scheduler = WarmupLR
[2021-03-31 09:49:12,590] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupLR object at 0x7fef53ed1d68>
[2021-03-31 09:49:12,591] [INFO] [logging.py:60:log_dist] [Rank 0] step=0, skipped=0, lr=[3e-05], mom=[[0.8, 0.999]]
[2021-03-31 09:49:12,593] [INFO] [config.py:737:print] DeepSpeedEngine configuration:
[2021-03-31 09:49:12,594] [INFO] [config.py:741:print]   activation_checkpointing_config  {
    "contiguous_memory_optimization": false,
    "cpu_checkpointing": false,
    "number_checkpoints": null,
    "partition_activations": false,
    "profile": false,
    "synchronize_checkpoint_boundary": false
}
[2021-03-31 09:49:12,594] [INFO] [config.py:741:print]   allreduce_always_fp32 ........ False
[2021-03-31 09:49:12,595] [INFO] [config.py:741:print]   amp_enabled .................. False
[2021-03-31 09:49:12,596] [INFO] [config.py:741:print]   amp_params ................... False
[2021-03-31 09:49:12,596] [INFO] [config.py:741:print]   checkpoint_tag_validation_enabled  True
[2021-03-31 09:49:12,597] [INFO] [config.py:741:print]   checkpoint_tag_validation_fail  False
[2021-03-31 09:49:12,600] [INFO] [config.py:741:print]   disable_allgather ............ False
[2021-03-31 09:49:12,601] [INFO] [config.py:741:print]   dump_state ................... False
[2021-03-31 09:49:12,602] [INFO] [config.py:741:print]   dynamic_loss_scale_args ...... {'init_scale': 4294967296, 'scale_window': 1000, 'delayed_shift': 2, 'min_scale': 1}
[2021-03-31 09:49:12,603] [INFO] [config.py:741:print]   elasticity_enabled ........... False
[2021-03-31 09:49:12,603] [INFO] [config.py:741:print]   flops_profiler_config ........ {
    "detailed": true,
    "enabled": false,
    "module_depth": -1,
    "profile_step": 1,
    "top_modules": 3
}
[2021-03-31 09:49:12,604] [INFO] [config.py:741:print]   fp16_enabled ................. True
[2021-03-31 09:49:12,605] [INFO] [config.py:741:print]   global_rank .................. 0
[2021-03-31 09:49:12,605] [INFO] [config.py:741:print]   gradient_accumulation_steps .. 1
[2021-03-31 09:49:12,606] [INFO] [config.py:741:print]   gradient_clipping ............ 1.0
[2021-03-31 09:49:12,607] [INFO] [config.py:741:print]   gradient_predivide_factor .... 1.0
[2021-03-31 09:49:12,607] [INFO] [config.py:741:print]   initial_dynamic_scale ........ 4294967296
[2021-03-31 09:49:12,608] [INFO] [config.py:741:print]   loss_scale ................... 0
[2021-03-31 09:49:12,609] [INFO] [config.py:741:print]   memory_breakdown ............. False
[2021-03-31 09:49:12,610] [INFO] [config.py:741:print]   optimizer_legacy_fusion ...... False
[2021-03-31 09:49:12,610] [INFO] [config.py:741:print]   optimizer_name ............... adamw
[2021-03-31 09:49:12,611] [INFO] [config.py:741:print]   optimizer_params ............. {'lr': 3e-05, 'betas': [0.8, 0.999], 'eps': 1e-08, 'weight_decay': 3e-07}
[2021-03-31 09:49:12,612] [INFO] [config.py:741:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2021-03-31 09:49:12,612] [INFO] [config.py:741:print]   pld_enabled .................. False
[2021-03-31 09:49:12,613] [INFO] [config.py:741:print]   pld_params ................... False
[2021-03-31 09:49:12,614] [INFO] [config.py:741:print]   prescale_gradients ........... False
[2021-03-31 09:49:12,616] [INFO] [config.py:741:print]   scheduler_name ............... WarmupLR
[2021-03-31 09:49:12,616] [INFO] [config.py:741:print]   scheduler_params ............. {'warmup_min_lr': 0, 'warmup_max_lr': 3e-05, 'warmup_num_steps': 500}
[2021-03-31 09:49:12,617] [INFO] [config.py:741:print]   sparse_attention ............. None
[2021-03-31 09:49:12,618] [INFO] [config.py:741:print]   sparse_gradients_enabled ..... False
[2021-03-31 09:49:12,618] [INFO] [config.py:741:print]   steps_per_print .............. 2000
[2021-03-31 09:49:12,619] [INFO] [config.py:741:print]   tensorboard_enabled .......... False
[2021-03-31 09:49:12,620] [INFO] [config.py:741:print]   tensorboard_job_name ......... DeepSpeedJobName
[2021-03-31 09:49:12,620] [INFO] [config.py:741:print]   tensorboard_output_path ...... 
[2021-03-31 09:49:12,621] [INFO] [config.py:741:print]   train_batch_size ............. 4
[2021-03-31 09:49:12,622] [INFO] [config.py:741:print]   train_micro_batch_size_per_gpu  4
[2021-03-31 09:49:12,622] [INFO] [config.py:741:print]   wall_clock_breakdown ......... False
[2021-03-31 09:49:12,623] [INFO] [config.py:741:print]   world_size ................... 1
[2021-03-31 09:49:12,624] [INFO] [config.py:741:print]   zero_allow_untested_optimizer  False
[2021-03-31 09:49:12,625] [INFO] [config.py:741:print]   zero_config .................. {
    "allgather_bucket_size": 150000000.0,
    "allgather_partitions": true,
    "contiguous_gradients": true,
    "cpu_offload": true,
    "cpu_offload_params": false,
    "cpu_offload_use_pin_memory": false,
    "elastic_checkpoint": true,
    "gather_fp16_weights_on_model_save": false,
    "load_from_fp32_weights": true,
    "max_live_parameters": 1000000000,
    "max_reuse_distance": 1000000000,
    "overlap_comm": true,
    "param_persistence_threshold": 100000,
    "prefetch_bucket_size": 50000000,
    "reduce_bucket_size": 150000000.0,
    "reduce_scatter": true,
    "stage": 2,
    "sub_group_size": 1000000000000
}
[2021-03-31 09:49:12,625] [INFO] [config.py:741:print]   zero_enabled ................. True
[2021-03-31 09:49:12,626] [INFO] [config.py:741:print]   zero_optimization_stage ...... 2
[2021-03-31 09:49:12,628] [INFO] [config.py:748:print]   json = {
    "fp16":{
        "enabled":true,
        "hysteresis":2,
        "loss_scale":0,
        "loss_scale_window":1000,
        "min_loss_scale":1
    },
    "gradient_accumulation_steps":1,
    "gradient_clipping":1.0,
    "optimizer":{
        "params":{
            "betas":[
                0.8,
                0.999
            ],
            "eps":1e-08,
            "lr":3e-05,
            "weight_decay":3e-07
        },
        "type":"AdamW"
    },
    "scheduler":{
        "params":{
            "warmup_max_lr":3e-05,
            "warmup_min_lr":0,
            "warmup_num_steps":500
        },
        "type":"WarmupLR"
    },
    "steps_per_print":2000,
    "train_micro_batch_size_per_gpu":4,
    "wall_clock_breakdown":false,
    "zero_optimization":{
        "allgather_bucket_size":150000000.0,
        "allgather_partitions":true,
        "contiguous_gradients":true,
        "cpu_offload":true,
        "overlap_comm":true,
        "reduce_bucket_size":150000000.0,
        "reduce_scatter":true,
        "stage":2
    }
}
```
But both GPUs were used if I convert my notebook to python script and run script with command - `!NCCL_SOCKET_IFNAME=lo deepspeed Deberta_V2_XXLarge.py --deepspeed ds_config.json`  and here's the log -
```
[2021-03-31 09:26:36,687] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
[2021-03-31 09:26:36,714] [INFO] [runner.py:358:main] cmd = /usr/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMV19 --master_addr=127.0.0.1 --master_port=29500 Deberta_V2_XXLarge.py --deepspeed ds_config.json
[2021-03-31 09:26:38,357] [INFO] [launch.py:73:main] 0 NCCL_SOCKET_IFNAME lo
[2021-03-31 09:26:38,357] [INFO] [launch.py:80:main] WORLD INFO DICT: {'localhost': [0, 1]}
[2021-03-31 09:26:38,357] [INFO] [launch.py:89:main] nnodes=1, num_local_procs=2, node_rank=0
[2021-03-31 09:26:38,357] [INFO] [launch.py:101:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1]})
[2021-03-31 09:26:38,358] [INFO] [launch.py:102:main] dist_world_size=2
[2021-03-31 09:26:38,358] [INFO] [launch.py:105:main] Setting CUDA_VISIBLE_DEVICES=0,1
2021-03-31 09:26:40.269001: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2021-03-31 09:26:40.269004: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
[2021-03-31 09:27:27,272] [INFO] [distributed.py:47:init_distributed] Initializing torch distributed with backend: nccl
[2021-03-31 09:27:31,495] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.13+7fcc891, git-hash=7fcc891, git-branch=master
[2021-03-31 09:27:32,428] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 2, parameter_parallel_size: 2
[2021-03-31 09:27:32,834] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 2, parameter_parallel_size: 2
Adam Optimizer #0 is created with AVX512 arithmetic capability.
Config: alpha=0.000030, betas=(0.800000, 0.999000), weight_decay=0.000000, adam_w=1
[2021-03-31 09:27:36,438] [INFO] [engine.py:608:_configure_optimizer] Using DeepSpeed Optimizer param name adamw as basic optimizer
[2021-03-31 09:27:36,438] [INFO] [engine.py:612:_configure_optimizer] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
[2021-03-31 09:27:36,439] [INFO] [logging.py:60:log_dist] [Rank 0] Creating fp16 ZeRO stage 2 optimizer
Adam Optimizer #0 is created with AVX512 arithmetic capability.
Config: alpha=0.000030, betas=(0.800000, 0.999000), weight_decay=0.000000, adam_w=1
[2021-03-31 09:27:36,441] [INFO] [stage2.py:130:__init__] Reduce bucket size 150000000.0
[2021-03-31 09:27:36,441] [INFO] [stage2.py:131:__init__] Allgather bucket size 150000000.0
[2021-03-31 09:27:36,441] [INFO] [stage2.py:132:__init__] CPU Offload: True
[2021-03-31 09:27:40,524] [INFO] [stage2.py:399:__init__] optimizer state initialized
[2021-03-31 09:27:40,530] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
[2021-03-31 09:27:40,533] [INFO] [engine.py:445:_configure_lr_scheduler] DeepSpeed using configured LR scheduler = WarmupLR
[2021-03-31 09:27:40,534] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupLR object at 0x7f105c6a0b00>
[2021-03-31 09:27:40,535] [INFO] [logging.py:60:log_dist] [Rank 0] step=0, skipped=0, lr=[3e-05], mom=[[0.8, 0.999]]
[2021-03-31 09:27:40,536] [INFO] [config.py:737:print] DeepSpeedEngine configuration:
[2021-03-31 09:27:40,537] [INFO] [config.py:741:print]   activation_checkpointing_config  {
    "contiguous_memory_optimization": false,
    "cpu_checkpointing": false,
    "number_checkpoints": null,
    "partition_activations": false,
    "profile": false,
    "synchronize_checkpoint_boundary": false
}
[2021-03-31 09:27:40,537] [INFO] [config.py:741:print]   allreduce_always_fp32 ........ False
[2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   amp_enabled .................. False
[2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   amp_params ................... False
[2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   checkpoint_tag_validation_enabled  True
[2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   checkpoint_tag_validation_fail  False
[2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   disable_allgather ............ False
[2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   dump_state ................... False
[2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   dynamic_loss_scale_args ...... {'init_scale': 4294967296, 'scale_window': 1000, 'delayed_shift': 2, 'min_scale': 1}
[2021-03-31 09:27:40,539] [INFO] [config.py:741:print]   elasticity_enabled ........... False
[2021-03-31 09:27:40,539] [INFO] [config.py:741:print]   flops_profiler_config ........ {
    "detailed": true,
    "enabled": false,
    "module_depth": -1,
    "profile_step": 1,
    "top_modules": 3
}
[2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   fp16_enabled ................. True
[2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   global_rank .................. 0
[2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   gradient_accumulation_steps .. 1
[2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   gradient_clipping ............ 1.0
[2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   gradient_predivide_factor .... 1.0
[2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   initial_dynamic_scale ........ 4294967296
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   loss_scale ................... 0
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   memory_breakdown ............. False
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   optimizer_legacy_fusion ...... False
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   optimizer_name ............... adamw
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   optimizer_params ............. {'lr': 3e-05, 'betas': [0.8, 0.999], 'eps': 1e-08, 'weight_decay': 3e-07}
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   pld_enabled .................. False
[2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   pld_params ................... False
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   prescale_gradients ........... False
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   scheduler_name ............... WarmupLR
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   scheduler_params ............. {'warmup_min_lr': 0, 'warmup_max_lr': 3e-05, 'warmup_num_steps': 500}
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   sparse_attention ............. None
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   sparse_gradients_enabled ..... False
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   steps_per_print .............. 2000
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   tensorboard_enabled .......... False
[2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   tensorboard_job_name ......... DeepSpeedJobName
[2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   tensorboard_output_path ...... 
[2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   train_batch_size ............. 8
[2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   train_micro_batch_size_per_gpu  4
[2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   wall_clock_breakdown ......... False
[2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   world_size ................... 2
[2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   zero_allow_untested_optimizer  False
[2021-03-31 09:27:40,544] [INFO] [config.py:741:print]   zero_config .................. {
    "allgather_bucket_size": 150000000.0,
    "allgather_partitions": true,
    "contiguous_gradients": true,
    "cpu_offload": true,
    "cpu_offload_params": false,
    "cpu_offload_use_pin_memory": false,
    "elastic_checkpoint": true,
    "gather_fp16_weights_on_model_save": false,
    "load_from_fp32_weights": true,
    "max_live_parameters": 1000000000,
    "max_reuse_distance": 1000000000,
    "overlap_comm": true,
    "param_persistence_threshold": 100000,
    "prefetch_bucket_size": 50000000,
    "reduce_bucket_size": 150000000.0,
    "reduce_scatter": true,
    "stage": 2,
    "sub_group_size": 1000000000000
}
[2021-03-31 09:27:40,544] [INFO] [config.py:741:print]   zero_enabled ................. True
[2021-03-31 09:27:40,545] [INFO] [config.py:741:print]   zero_optimization_stage ...... 2
[2021-03-31 09:27:40,546] [INFO] [config.py:748:print]   json = {
    "fp16":{
        "enabled":true,
        "hysteresis":2,
        "loss_scale":0,
        "loss_scale_window":1000,
        "min_loss_scale":1
    },
    "gradient_accumulation_steps":1,
    "gradient_clipping":1.0,
    "optimizer":{
        "params":{
            "betas":[
                0.8,
                0.999
            ],
            "eps":1e-08,
            "lr":3e-05,
            "weight_decay":3e-07
        },
        "type":"AdamW"
    },
    "scheduler":{
        "params":{
            "warmup_max_lr":3e-05,
            "warmup_min_lr":0,
            "warmup_num_steps":500
        },
        "type":"WarmupLR"
    },
    "steps_per_print":2000,
    "train_micro_batch_size_per_gpu":4,
    "wall_clock_breakdown":false,
    "zero_optimization":{
        "allgather_bucket_size":150000000.0,
        "allgather_partitions":true,
        "contiguous_gradients":true,
        "cpu_offload":true,
        "overlap_comm":true,
        "reduce_bucket_size":150000000.0,
        "reduce_scatter":true,
        "stage":2
    }
}
```

Plz suggest whether I need to make some changes..

Verions I'm using - 
```
torch-1.7.1+cu101
transformers-4.4.2
deepspeed-0.3.13
```
**Who can help**
@LysandreJik 
@stas00 


Comments:
That's correct. 

You need a separate process for each gpu under DeepSpeed for communications to work. I will update the docs to make this clear.

If you want to use multiple gpus you must use the launcher. So you can still use the notebook to set things up, but the training must happen in external process, e.g. see:

https://github.com/stas00/porting/blob/master/transformers/deepspeed/DeepSpeed_on_colab_CLI.ipynb

but edit the launcher line to use `deepspeed --num_gpus 2`

I will close this for now as it's a clear "this is not possible due to the DeepSpeed design", but if you have some further questions please don't hesitate to follow up.