Repository: microsoft/DeepSpeed

Issue:
[BUG] ValueError: max() arg is an empty sequence using bf16 zero stage3
```
│ /opt/conda/lib/python3.7/site-packages/deepspeed/runtime/zero/stage3.py:307  │
│ in <listcomp>                                                                │
│                                                                              │
│    304 │   │   │   max([                                                     │
│    305 │   │   │   │   max(tensor.numel(),                                   │
│    306 │   │   │   │   │   tensor.ds_numel) for tensor in fp16_partitioned_g │
│ ❱  307 │   │   │   ]) for fp16_partitioned_group in self.fp16_partitioned_gr │
│    308 │   │   ])                                                            │
│    309 │   │   print_rank_0(                                                 │
│    310 │   │   │   f'Largest partitioned param numel = {largest_partitioned_ │
╰──────────────────────────────────────────────────────────────────────────────╯
ValueError: max() arg is an empty sequence
```
**To Reproduce**
Steps to reproduce the behavior:
Happened during finetuning on flan 11b model . Here is the entire error gist - https://gist.github.com/sujithjoseph/c410514acfccc76974a8130a8afd2169 

Here is the deepspeed config https://gist.github.com/sujithjoseph/92bf27de6bba704b57c3b9eb7aa00365



**ds_report output**
ds report - https://gist.github.com/sujithjoseph/c725de5fb38bb3c20e4fb6fd55f63848 


**System info (please complete the following information):**
 - OS: Debian GNU/Linux 10 (buster)
 - GPU count and types [ 1 machine with 4 A100s - 40G*4]
 - Python version 3.7

**Launcher context**
Are you launching your experiment with the `deepspeed` launcher, MPI, or something else? Accelerate + PEFT


deepspeed_config:
--
deepspeed_config_file: zero_stage3_offload_config.json
zero3_init_flag: true

**Additional context**

I assume that bf16 configs and fp16 configs are interchangeable 

```
    "bf16": {
        "enabled": true,
        "loss_scale": 0,
        "loss_scale_window": 1000,
        "initial_scale_power": 16,
        "hysteresis": 2,
        "min_loss_scale": 1
    }

```



Discussion:
Error also appears with fp16 instead of bf16 in deepspeed config and with zero3_init_flag: false in accelerate config with deepspeed as well. 
With Stage2 and no offsets , Get a different error 
```
 /opt/conda/lib/python3.7/site-packages/deepspeed/runtime/zero/stage_1_and_2. │
│ py:323 in __init__                                                           │
│                                                                              │
│    320 │   │   │   │   self.flatten_dense_tensors_aligned(                   │
│    321 │   │   │   │   │   self.round_robin_bit16_groups[i],                 │
│    322 │   │   │   │   │   self.nccl_start_alignment_factor *                │
│ ❱  323 │   │   │   │   │   dist.get_world_size(group=self.real_dp_process_gr │
│    324 │   │   │   │   │   │   torch.cuda.current_device()))                 │
│    325 │   │   │   see_memory_usage(f"After flattening and moving param grou │
│    326 │   │   │   │   │   │   │    force=False)                             │
│                                                                              │
│ /opt/conda/lib/python3.7/site-packages/deepspeed/runtime/zero/stage_1_and_2. │
│ py:862 in flatten_dense_tensors_aligned                                      │
│                                                                              │
│    859 │                                                                     │
│    860 │   # create a flat tensor aligned at the alignment boundary          │
│    861 │   def flatten_dense_tensors_aligned(self, tensor_list, alignment):  │
│ ❱  862 │   │   return self.flatten(align_dense_tensors(tensor_list, alignmen │
│    863 │                                                                     │
│    864 │   ############### Independent Partition Gradient ################## │
│    865 │   def reduce_independent_p_g_buckets_and_remove_grads(self, param,  │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: torch.cat(): expected a non-empty list of Tensors
```
Could this be an issue with dataset?
Was able to sort it out using the below accelerate + DS config.  Now dealing with an OOM issue, but not sure why the previous DeepSpeed config didnt work
```
compute_environment: LOCAL_MACHINE
deepspeed_config:
  gradient_accumulation_steps: 1
  gradient_clipping: 1.0
  offload_optimizer_device: cpu
  offload_param_device: cpu
  zero3_init_flag: true
  zero3_save_16bit_model: true
  zero_stage: 3
distributed_type: DEEPSPEED
downcast_bf16: true
dynamo_backend: 'NO'
fsdp_config: {}
machine_rank: 0
main_training_function: main
megatron_lm_config: {}
mixed_precision: 'bf16'
num_machines: 1
num_processes: 4
rdzv_backend: static
same_network: true
use_cpu: false
```
How can we estimate the # of GPUs Needed (each with 40 GB) . for flan-t5-11b with cpu param / optimizer offloading , 0.49GB | offload_param=cpu , offload_optimizer=cpu , zero_init=1 is what the estimate is. 

Estimated memory needed for params, optim states and gradients for a:
HW: Setup with 1 node, 4 GPUs per node.
SW: Model with 11003M total params, 131M largest layer params.
  per CPU  |  per GPU |   Options
  276.70GB |   0.49GB | offload_param=cpu , offload_optimizer=cpu , zero_init=1
  276.70GB |   0.49GB | offload_param=cpu , offload_optimizer=cpu , zero_init=0
  245.95GB |   5.61GB | offload_param=none, offload_optimizer=cpu , zero_init=1
  245.95GB |   5.61GB | offload_param=none, offload_optimizer=cpu , zero_init=0
    2.94GB |  46.61GB | offload_param=none, offload_optimizer=none, zero_init=1
  245.95GB |  46.61GB | offload_param=none, offload_optimizer=none, zero_init=0
With Batch size as 1, it works without OOM. How can we estimate the # of GPUs needed for batchsize of 4 or 8, without trial and error. With batch size as 1, I see only 26903MiB used per GPU max. For batch size as 2, It works for some time (3-4 hours) with 8 40 GB GPUs with almost all 40 GB utilized and then goes into OOM. How can I cap the GPU memory used?
With the following deepspeed config 
```
deepspeed_config:
  gradient_accumulation_steps: 2
  gradient_clipping: 1.0
  offload_optimizer_device: cpu
  offload_param_device: cpu
  zero3_init_flag: true
  zero3_save_16bit_model: true
  zero_stage: 3
  bf16:enabled: true
``` 
and 
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.allow_tf32 = True
set in the code, would deepspeed use tf32 or bf16 ?
@sujithjoseph, deepspeed should use bf16. Are you observing something different? 
@tjruwase , it did work with bf16. The only Question I have is Can i use max memory to restrict the memory used by the model during fine-tuning like the below one used for inference etc. 

max_memory={0: "25GIB", "cpu":"120GB"}


model = load_checkpoint_and_dispatch(
    model, model_id, device_map="auto", max_memory=max_memory, no_split_module_classes=["T5Block"]
)
Got it. I don't have experience with those memory restriction flags, which seem to be Accelerate flags. I don't think those flags are hooked into deepspeed. Can you please pose this question on their forum? I think we can work with them to enable the desired feature. 
@sujithjoseph I faced the same issues as your memtioned above, both issue in stage 2 and stage 3 were the same to you. Did you have any workaround for this?
Ran into exact same error when running DeepSpeed on Ray. Following this thread.
same error RuntimeError: torch.cat(): expected a non-empty list of Tensors when accelerate.prepare. So how to solve it?
@zhenlohuang, @shaowei-su, @SupetZYK, it seems that @sujithjoseph resolved the original issue with the following
https://github.com/microsoft/DeepSpeed/issues/2820#issuecomment-1427255633

If the workaround does not work for you, please open a new issue and share details to help us repro. Thanks! 
@tjruwase I was able to run DS + stage 3 + fp16 by disabling `optimizer` section in the DS config, which I found negative impacts on the model quality.

If I switch to DS + stage 2, then it's the same runtime error @SupetZYK posted above.
```
  File "/home/default_user/.conda/envs/user/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1298, in _configure_optimizer
    self.optimizer = self._configure_zero_optimizer(basic_optimizer)
  File "/home/default_user/.conda/envs/user/lib/python3.10/site-packages/deepspeed/runtime/engine.py", line 1547, in _configure_zero_optimizer
    optimizer = DeepSpeedZeroOptimizer(
  File "/home/default_user/.conda/envs/user/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 324, in __init__
    self.flatten_dense_tensors_aligned(
  File "/home/default_user/.conda/envs/user/lib/python3.10/site-packages/deepspeed/runtime/zero/stage_1_and_2.py", line 867, in flatten_dense_tensors_aligned
    return self.flatten(align_dense_tensors(tensor_list, alignment))
RuntimeError: torch.cat(): expected a non-empty list of Tensors
```
@shaowei-su and @SupetZYK, it seems you are both seeing a different error from the original posting. Can you please open a new issue and share details for repro? I will close this in the meantime. Thanks! 
same error here,any update?
FWIW I got this error when I put my model accidentally on inference mode. 
My peft config had `inference_mode: True`.
Same error when using loralib with zero2 & 3
@bestpredicts and @Wesley-Jzy, are you able to provide repro steps?

Resolving patch:
diff --git a/deepspeed/runtime/engine.py b/deepspeed/runtime/engine.py
--- a/deepspeed/runtime/engine.py
+++ b/deepspeed/runtime/engine.py
@@ -1181,13 +1181,11 @@ def _do_optimizer_sanity_check(self, basic_optimizer):
 
     # Configure optimizer
     def _configure_optimizer(self, client_optimizer, model_parameters):
-        if client_optimizer is not None:
+        if client_optimizer is None:
+            basic_optimizer = self._configure_basic_optimizer(model_parameters)
+            log_dist(f"Using DeepSpeed Optimizer param name {self.optimizer_name()} as basic optimizer", ranks=[0])
+        else:
             if isinstance(client_optimizer, tuple(self._supported_optims())):
-                client_optimizer.param_groups[:] = [
-                    pg for pg in client_optimizer.param_groups if len(pg["params"]) != 0
-                ]
-                log_dist("Removing param_group that has no 'params' in the client Optimizer", ranks=[0])
-
                 basic_optimizer = client_optimizer
                 log_dist('Using client Optimizer as basic optimizer', ranks=[0])
             else:
@@ -1198,9 +1196,9 @@ def _configure_optimizer(self, client_optimizer, model_parameters):
                 if self.zero_force_ds_cpu_optimizer():
                     msg = f'You are using ZeRO-Offload with a client provided optimizer ({type(basic_optimizer)}) which in most cases will yield poor performance. Please either use deepspeed.ops.adam.DeepSpeedCPUAdam or set an optimizer in your ds-config (https://www.deepspeed.ai/docs/config-json/#optimizer-parameters). If you really want to use a custom optimizer w. ZeRO-Offload and understand the performance impacts you can also set <"zero_force_ds_cpu_optimizer": false> in your configuration file.'
                     raise ZeRORuntimeException(msg)
-        else:
-            basic_optimizer = self._configure_basic_optimizer(model_parameters)
-            log_dist(f"Using DeepSpeed Optimizer param name {self.optimizer_name()} as basic optimizer", ranks=[0])
+
+        basic_optimizer.param_groups[:] = [pg for pg in basic_optimizer.param_groups if len(pg["params"]) != 0]
+        log_dist("Removing param_group that has no 'params' in the basic Optimizer", ranks=[0])
 
         self._check_for_duplicates(basic_optimizer)
 
diff --git a/deepspeed/runtime/zero/stage3.py b/deepspeed/runtime/zero/stage3.py
--- a/deepspeed/runtime/zero/stage3.py
+++ b/deepspeed/runtime/zero/stage3.py
@@ -415,9 +415,20 @@ def initialize_ds_offload(
 
     def _get_trainable_parameter_groups(self):
         param_groups = []
+        PARAMS_KEY = "params"
         for param_group in self.optimizer.param_groups:
-            trainable_params = {"params": [p for p in param_group["params"] if p.requires_grad]}
-            param_groups.append(trainable_params)
+            trainable_params = [p for p in param_group[PARAMS_KEY] if p.requires_grad]
+            if len(trainable_params) == 0:
+                continue
+
+            trainable_param_group = {}
+            for key in param_group.keys():
+                if key == PARAMS_KEY:
+                    trainable_param_group[PARAMS_KEY] = trainable_params
+                else:
+                    trainable_param_group[key] = param_group[key]
+            param_groups.append(trainable_param_group)
+
         return param_groups
 
     def _set_zero_group_parallelism(self):


Test patch:
diff --git a/tests/unit/runtime/zero/test_zero.py b/tests/unit/runtime/zero/test_zero.py
--- a/tests/unit/runtime/zero/test_zero.py
+++ b/tests/unit/runtime/zero/test_zero.py
@@ -1472,3 +1472,53 @@ def test_training_partition_cache(self, training):
 
         model.empty_partition_cache()
         assert sum([p.numel() for p in model.parameters()]) == 0
+
+
+@pytest.mark.parametrize("use_client_optimizer", [True, False])
+@pytest.mark.parametrize("empty_weight_group", [True, False])
+@pytest.mark.parametrize("dtype", [torch.bfloat16, torch.float16, torch.float32])
+class TestEmptyParameterGroup(DistributedTest):
+    world_size = 1
+
+    def test_empty_param_groups(self, dtype, use_client_optimizer, empty_weight_group):
+        model = SimpleModel(hidden_dim=4, nlayers=4)
+        param_groups = [
+            {
+                "params": [] if empty_weight_group else [l.weight for l in model.linears],
+                "weight_decay": 0.01,
+            },
+            {
+                "params": [l.bias for l in model.linears] if empty_weight_group else [],
+                "weight_decay": 0.0
+            },
+        ]
+
+        config_dict = {
+            "train_micro_batch_size_per_gpu": 1,
+            "steps_per_print": 1,
+            "zero_optimization": {
+                "stage": 3,
+                "stage3_param_persistence_threshold": 0,
+            },
+            "fp16": {
+                "enabled": dtype == torch.float16,
+            },
+            "bf16": {
+                "enabled": dtype == torch.bfloat16
+            }
+        }
+
+        if use_client_optimizer:
+            optimizer = deepspeed.ops.adam.FusedAdam(param_groups, lr=0.1)
+            model_parameters = model.parameters()
+        else:
+            config_dict["optimizer"] = {"type": "adamw"}
+            optimizer = None
+            model_parameters = param_groups
+
+        model, _, _, _ = deepspeed.initialize(
+            model=model,
+            model_parameters=model_parameters,
+            optimizer=optimizer,
+            config=config_dict,
+        )