text: 
  
  TITLE
  unable to use multiple GPUs with HF integration of DeepSpeed on Jupyter notebooks
  
  COMMENTS
  1
  
  REACTIONS
  +1: 0
  -1: 0
  laugh: 0
  hooray: 0
  heart: 0
  rocket: 0
  eyes: 0
  
  BODY
  Hi ,
  
  I'm using HF integration of DeepSpeed in my Jupyter Notebook by setting following env variables as suggested [here](https://huggingface.co/transformers/main_classes/trainer.html#deployment-in-notebooks)
  
  ```
  os.environ['MASTER_ADDR'] = 'localhost'
  os.environ['MASTER_PORT'] = '9889'
  os.environ['RANK'] = "0"
  os.environ['LOCAL_RANK'] = "0"
  os.environ['WORLD_SIZE'] = "1"
  os.environ['NCCL_SOCKET_IFNAME'] = 'lo'   ##because of my kubeflow setup
  ```
  with this setup I'm unable to utilize both GPU's that I have and here's the log info before it starts training- 
  ```
  [2021-03-31 09:49:02,510] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.13+7fcc891, git-hash=7fcc891, git-branch=master
  [2021-03-31 09:49:02,540] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 1, parameter_parallel_size: 1
  [2021-03-31 09:49:05,758] [INFO] [engine.py:608:_configure_optimizer] Using DeepSpeed Optimizer param name adamw as basic optimizer
  [2021-03-31 09:49:05,760] [INFO] [engine.py:612:_configure_optimizer] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
  Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
  [2021-03-31 09:49:05,761] [INFO] [logging.py:60:log_dist] [Rank 0] Creating fp16 ZeRO stage 2 optimizer
  [2021-03-31 09:49:05,764] [INFO] [stage2.py:130:__init__] Reduce bucket size 150000000.0
  [2021-03-31 09:49:05,765] [INFO] [stage2.py:131:__init__] Allgather bucket size 150000000.0
  [2021-03-31 09:49:05,766] [INFO] [stage2.py:132:__init__] CPU Offload: True
  [2021-03-31 09:49:12,585] [INFO] [stage2.py:399:__init__] optimizer state initialized
  [2021-03-31 09:49:12,588] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
  [2021-03-31 09:49:12,589] [INFO] [engine.py:445:_configure_lr_scheduler] DeepSpeed using configured LR scheduler = WarmupLR
  [2021-03-31 09:49:12,590] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupLR object at 0x7fef53ed1d68>
  [2021-03-31 09:49:12,591] [INFO] [logging.py:60:log_dist] [Rank 0] step=0, skipped=0, lr=[3e-05], mom=[[0.8, 0.999]]
  [2021-03-31 09:49:12,593] [INFO] [config.py:737:print] DeepSpeedEngine configuration:
  [2021-03-31 09:49:12,594] [INFO] [config.py:741:print]   activation_checkpointing_config  {
      "contiguous_memory_optimization": false,
      "cpu_checkpointing": false,
      "number_checkpoints": null,
      "partition_activations": false,
      "profile": false,
      "synchronize_checkpoint_boundary": false
  }
  [2021-03-31 09:49:12,594] [INFO] [config.py:741:print]   allreduce_always_fp32 ........ False
  [2021-03-31 09:49:12,595] [INFO] [config.py:741:print]   amp_enabled .................. False
  [2021-03-31 09:49:12,596] [INFO] [config.py:741:print]   amp_params ................... False
  [2021-03-31 09:49:12,596] [INFO] [config.py:741:print]   checkpoint_tag_validation_enabled  True
  [2021-03-31 09:49:12,597] [INFO] [config.py:741:print]   checkpoint_tag_validation_fail  False
  [2021-03-31 09:49:12,600] [INFO] [config.py:741:print]   disable_allgather ............ False
  [2021-03-31 09:49:12,601] [INFO] [config.py:741:print]   dump_state ................... False
  [2021-03-31 09:49:12,602] [INFO] [config.py:741:print]   dynamic_loss_scale_args ...... {'init_scale': 4294967296, 'scale_window': 1000, 'delayed_shift': 2, 'min_scale': 1}
  [2021-03-31 09:49:12,603] [INFO] [config.py:741:print]   elasticity_enabled ........... False
  [2021-03-31 09:49:12,603] [INFO] [config.py:741:print]   flops_profiler_config ........ {
      "detailed": true,
      "enabled": false,
      "module_depth": -1,
      "profile_step": 1,
      "top_modules": 3
  }
  [2021-03-31 09:49:12,604] [INFO] [config.py:741:print]   fp16_enabled ................. True
  [2021-03-31 09:49:12,605] [INFO] [config.py:741:print]   global_rank .................. 0
  [2021-03-31 09:49:12,605] [INFO] [config.py:741:print]   gradient_accumulation_steps .. 1
  [2021-03-31 09:49:12,606] [INFO] [config.py:741:print]   gradient_clipping ............ 1.0
  [2021-03-31 09:49:12,607] [INFO] [config.py:741:print]   gradient_predivide_factor .... 1.0
  [2021-03-31 09:49:12,607] [INFO] [config.py:741:print]   initial_dynamic_scale ........ 4294967296
  [2021-03-31 09:49:12,608] [INFO] [config.py:741:print]   loss_scale ................... 0
  [2021-03-31 09:49:12,609] [INFO] [config.py:741:print]   memory_breakdown ............. False
  [2021-03-31 09:49:12,610] [INFO] [config.py:741:print]   optimizer_legacy_fusion ...... False
  [2021-03-31 09:49:12,610] [INFO] [config.py:741:print]   optimizer_name ............... adamw
  [2021-03-31 09:49:12,611] [INFO] [config.py:741:print]   optimizer_params ............. {'lr': 3e-05, 'betas': [0.8, 0.999], 'eps': 1e-08, 'weight_decay': 3e-07}
  [2021-03-31 09:49:12,612] [INFO] [config.py:741:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
  [2021-03-31 09:49:12,612] [INFO] [config.py:741:print]   pld_enabled .................. False
  [2021-03-31 09:49:12,613] [INFO] [config.py:741:print]   pld_params ................... False
  [2021-03-31 09:49:12,614] [INFO] [config.py:741:print]   prescale_gradients ........... False
  [2021-03-31 09:49:12,616] [INFO] [config.py:741:print]   scheduler_name ............... WarmupLR
  [2021-03-31 09:49:12,616] [INFO] [config.py:741:print]   scheduler_params ............. {'warmup_min_lr': 0, 'warmup_max_lr': 3e-05, 'warmup_num_steps': 500}
  [2021-03-31 09:49:12,617] [INFO] [config.py:741:print]   sparse_attention ............. None
  [2021-03-31 09:49:12,618] [INFO] [config.py:741:print]   sparse_gradients_enabled ..... False
  [2021-03-31 09:49:12,618] [INFO] [config.py:741:print]   steps_per_print .............. 2000
  [2021-03-31 09:49:12,619] [INFO] [config.py:741:print]   tensorboard_enabled .......... False
  [2021-03-31 09:49:12,620] [INFO] [config.py:741:print]   tensorboard_job_name ......... DeepSpeedJobName
  [2021-03-31 09:49:12,620] [INFO] [config.py:741:print]   tensorboard_output_path ...... 
  [2021-03-31 09:49:12,621] [INFO] [config.py:741:print]   train_batch_size ............. 4
  [2021-03-31 09:49:12,622] [INFO] [config.py:741:print]   train_micro_batch_size_per_gpu  4
  [2021-03-31 09:49:12,622] [INFO] [config.py:741:print]   wall_clock_breakdown ......... False
  [2021-03-31 09:49:12,623] [INFO] [config.py:741:print]   world_size ................... 1
  [2021-03-31 09:49:12,624] [INFO] [config.py:741:print]   zero_allow_untested_optimizer  False
  [2021-03-31 09:49:12,625] [INFO] [config.py:741:print]   zero_config .................. {
      "allgather_bucket_size": 150000000.0,
      "allgather_partitions": true,
      "contiguous_gradients": true,
      "cpu_offload": true,
      "cpu_offload_params": false,
      "cpu_offload_use_pin_memory": false,
      "elastic_checkpoint": true,
      "gather_fp16_weights_on_model_save": false,
      "load_from_fp32_weights": true,
      "max_live_parameters": 1000000000,
      "max_reuse_distance": 1000000000,
      "overlap_comm": true,
      "param_persistence_threshold": 100000,
      "prefetch_bucket_size": 50000000,
      "reduce_bucket_size": 150000000.0,
      "reduce_scatter": true,
      "stage": 2,
      "sub_group_size": 1000000000000
  }
  [2021-03-31 09:49:12,625] [INFO] [config.py:741:print]   zero_enabled ................. True
  [2021-03-31 09:49:12,626] [INFO] [config.py:741:print]   zero_optimization_stage ...... 2
  [2021-03-31 09:49:12,628] [INFO] [config.py:748:print]   json = {
      "fp16":{
          "enabled":true,
          "hysteresis":2,
          "loss_scale":0,
          "loss_scale_window":1000,
          "min_loss_scale":1
      },
      "gradient_accumulation_steps":1,
      "gradient_clipping":1.0,
      "optimizer":{
          "params":{
              "betas":[
                  0.8,
                  0.999
              ],
              "eps":1e-08,
              "lr":3e-05,
              "weight_decay":3e-07
          },
          "type":"AdamW"
      },
      "scheduler":{
          "params":{
              "warmup_max_lr":3e-05,
              "warmup_min_lr":0,
              "warmup_num_steps":500
          },
          "type":"WarmupLR"
      },
      "steps_per_print":2000,
      "train_micro_batch_size_per_gpu":4,
      "wall_clock_breakdown":false,
      "zero_optimization":{
          "allgather_bucket_size":150000000.0,
          "allgather_partitions":true,
          "contiguous_gradients":true,
          "cpu_offload":true,
          "overlap_comm":true,
          "reduce_bucket_size":150000000.0,
          "reduce_scatter":true,
          "stage":2
      }
  }
  ```
  But both GPUs were used if I convert my notebook to python script and run script with command - `!NCCL_SOCKET_IFNAME=lo deepspeed Deberta_V2_XXLarge.py --deepspeed ds_config.json`  and here's the log -
  ```
  [2021-03-31 09:26:36,687] [WARNING] [runner.py:117:fetch_hostfile] Unable to find hostfile, will proceed with training with local resources only.
  [2021-03-31 09:26:36,714] [INFO] [runner.py:358:main] cmd = /usr/bin/python3 -u -m deepspeed.launcher.launch --world_info=eyJsb2NhbGhvc3QiOiBbMCwgMV19 --master_addr=127.0.0.1 --master_port=29500 Deberta_V2_XXLarge.py --deepspeed ds_config.json
  [2021-03-31 09:26:38,357] [INFO] [launch.py:73:main] 0 NCCL_SOCKET_IFNAME lo
  [2021-03-31 09:26:38,357] [INFO] [launch.py:80:main] WORLD INFO DICT: {'localhost': [0, 1]}
  [2021-03-31 09:26:38,357] [INFO] [launch.py:89:main] nnodes=1, num_local_procs=2, node_rank=0
  [2021-03-31 09:26:38,357] [INFO] [launch.py:101:main] global_rank_mapping=defaultdict(<class 'list'>, {'localhost': [0, 1]})
  [2021-03-31 09:26:38,358] [INFO] [launch.py:102:main] dist_world_size=2
  [2021-03-31 09:26:38,358] [INFO] [launch.py:105:main] Setting CUDA_VISIBLE_DEVICES=0,1
  2021-03-31 09:26:40.269001: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
  2021-03-31 09:26:40.269004: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
  [2021-03-31 09:27:27,272] [INFO] [distributed.py:47:init_distributed] Initializing torch distributed with backend: nccl
  [2021-03-31 09:27:31,495] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed info: version=0.3.13+7fcc891, git-hash=7fcc891, git-branch=master
  [2021-03-31 09:27:32,428] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 2, parameter_parallel_size: 2
  [2021-03-31 09:27:32,834] [INFO] [engine.py:80:_initialize_parameter_parallel_groups] data_parallel_size: 2, parameter_parallel_size: 2
  Adam Optimizer #0 is created with AVX512 arithmetic capability.
  Config: alpha=0.000030, betas=(0.800000, 0.999000), weight_decay=0.000000, adam_w=1
  [2021-03-31 09:27:36,438] [INFO] [engine.py:608:_configure_optimizer] Using DeepSpeed Optimizer param name adamw as basic optimizer
  [2021-03-31 09:27:36,438] [INFO] [engine.py:612:_configure_optimizer] DeepSpeed Basic Optimizer = DeepSpeedCPUAdam
  Checking ZeRO support for optimizer=DeepSpeedCPUAdam type=<class 'deepspeed.ops.adam.cpu_adam.DeepSpeedCPUAdam'>
  [2021-03-31 09:27:36,439] [INFO] [logging.py:60:log_dist] [Rank 0] Creating fp16 ZeRO stage 2 optimizer
  Adam Optimizer #0 is created with AVX512 arithmetic capability.
  Config: alpha=0.000030, betas=(0.800000, 0.999000), weight_decay=0.000000, adam_w=1
  [2021-03-31 09:27:36,441] [INFO] [stage2.py:130:__init__] Reduce bucket size 150000000.0
  [2021-03-31 09:27:36,441] [INFO] [stage2.py:131:__init__] Allgather bucket size 150000000.0
  [2021-03-31 09:27:36,441] [INFO] [stage2.py:132:__init__] CPU Offload: True
  [2021-03-31 09:27:40,524] [INFO] [stage2.py:399:__init__] optimizer state initialized
  [2021-03-31 09:27:40,530] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed Final Optimizer = adamw
  [2021-03-31 09:27:40,533] [INFO] [engine.py:445:_configure_lr_scheduler] DeepSpeed using configured LR scheduler = WarmupLR
  [2021-03-31 09:27:40,534] [INFO] [logging.py:60:log_dist] [Rank 0] DeepSpeed LR Scheduler = <deepspeed.runtime.lr_schedules.WarmupLR object at 0x7f105c6a0b00>
  [2021-03-31 09:27:40,535] [INFO] [logging.py:60:log_dist] [Rank 0] step=0, skipped=0, lr=[3e-05], mom=[[0.8, 0.999]]
  [2021-03-31 09:27:40,536] [INFO] [config.py:737:print] DeepSpeedEngine configuration:
  [2021-03-31 09:27:40,537] [INFO] [config.py:741:print]   activation_checkpointing_config  {
      "contiguous_memory_optimization": false,
      "cpu_checkpointing": false,
      "number_checkpoints": null,
      "partition_activations": false,
      "profile": false,
      "synchronize_checkpoint_boundary": false
  }
  [2021-03-31 09:27:40,537] [INFO] [config.py:741:print]   allreduce_always_fp32 ........ False
  [2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   amp_enabled .................. False
  [2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   amp_params ................... False
  [2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   checkpoint_tag_validation_enabled  True
  [2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   checkpoint_tag_validation_fail  False
  [2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   disable_allgather ............ False
  [2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   dump_state ................... False
  [2021-03-31 09:27:40,538] [INFO] [config.py:741:print]   dynamic_loss_scale_args ...... {'init_scale': 4294967296, 'scale_window': 1000, 'delayed_shift': 2, 'min_scale': 1}
  [2021-03-31 09:27:40,539] [INFO] [config.py:741:print]   elasticity_enabled ........... False
  [2021-03-31 09:27:40,539] [INFO] [config.py:741:print]   flops_profiler_config ........ {
      "detailed": true,
      "enabled": false,
      "module_depth": -1,
      "profile_step": 1,
      "top_modules": 3
  }
  [2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   fp16_enabled ................. True
  [2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   global_rank .................. 0
  [2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   gradient_accumulation_steps .. 1
  [2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   gradient_clipping ............ 1.0
  [2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   gradient_predivide_factor .... 1.0
  [2021-03-31 09:27:40,540] [INFO] [config.py:741:print]   initial_dynamic_scale ........ 4294967296
  [2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   loss_scale ................... 0
  [2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   memory_breakdown ............. False
  [2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   optimizer_legacy_fusion ...... False
  [2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   optimizer_name ............... adamw
  [2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   optimizer_params ............. {'lr': 3e-05, 'betas': [0.8, 0.999], 'eps': 1e-08, 'weight_decay': 3e-07}
  [2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
  [2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   pld_enabled .................. False
  [2021-03-31 09:27:40,541] [INFO] [config.py:741:print]   pld_params ................... False
  [2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   prescale_gradients ........... False
  [2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   scheduler_name ............... WarmupLR
  [2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   scheduler_params ............. {'warmup_min_lr': 0, 'warmup_max_lr': 3e-05, 'warmup_num_steps': 500}
  [2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   sparse_attention ............. None
  [2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   sparse_gradients_enabled ..... False
  [2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   steps_per_print .............. 2000
  [2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   tensorboard_enabled .......... False
  [2021-03-31 09:27:40,542] [INFO] [config.py:741:print]   tensorboard_job_name ......... DeepSpeedJobName
  [2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   tensorboard_output_path ...... 
  [2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   train_batch_size ............. 8
  [2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   train_micro_batch_size_per_gpu  4
  [2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   wall_clock_breakdown ......... False
  [2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   world_size ................... 2
  [2021-03-31 09:27:40,543] [INFO] [config.py:741:print]   zero_allow_untested_optimizer  False
  [2021-03-31 09:27:40,544] [INFO] [config.py:741:print]   zero_config .................. {
      "allgather_bucket_size": 150000000.0,
      "allgather_partitions": true,
      "contiguous_gradients": true,
      "cpu_offload": true,
      "cpu_offload_params": false,
      "cpu_offload_use_pin_memory": false,
      "elastic_checkpoint": true,
      "gather_fp16_weights_on_model_save": false,
      "load_from_fp32_weights": true,
      "max_live_parameters": 1000000000,
      "max_reuse_distance": 1000000000,
      "overlap_comm": true,
      "param_persistence_threshold": 100000,
      "prefetch_bucket_size": 50000000,
      "reduce_bucket_size": 150000000.0,
      "reduce_scatter": true,
      "stage": 2,
      "sub_group_size": 1000000000000
  }
  [2021-03-31 09:27:40,544] [INFO] [config.py:741:print]   zero_enabled ................. True
  [2021-03-31 09:27:40,545] [INFO] [config.py:741:print]   zero_optimization_stage ...... 2
  [2021-03-31 09:27:40,546] [INFO] [config.py:748:print]   json = {
      "fp16":{
          "enabled":true,
          "hysteresis":2,
          "loss_scale":0,
          "loss_scale_window":1000,
          "min_loss_scale":1
      },
      "gradient_accumulation_steps":1,
      "gradient_clipping":1.0,
      "optimizer":{
          "params":{
              "betas":[
                  0.8,
                  0.999
              ],
              "eps":1e-08,
              "lr":3e-05,
              "weight_decay":3e-07
          },
          "type":"AdamW"
      },
      "scheduler":{
          "params":{
              "warmup_max_lr":3e-05,
              "warmup_min_lr":0,
              "warmup_num_steps":500
          },
          "type":"WarmupLR"
      },
      "steps_per_print":2000,
      "train_micro_batch_size_per_gpu":4,
      "wall_clock_breakdown":false,
      "zero_optimization":{
          "allgather_bucket_size":150000000.0,
          "allgather_partitions":true,
          "contiguous_gradients":true,
          "cpu_offload":true,
          "overlap_comm":true,
          "reduce_bucket_size":150000000.0,
          "reduce_scatter":true,
          "stage":2
      }
  }
  ```
  
  Plz suggest whether I need to make some changes..
  
  Verions I'm using - 
  ```
  torch-1.7.1+cu101
  transformers-4.4.2
  deepspeed-0.3.13
  ```
  **Who can help**
  @LysandreJik 
  @stas00 
  