Repository: speechbrain/speechbrain

Issue:
How to specify distributed GPU numbers
Hi:

I want to ask if there any way to specify distributed GPU numbers

I used the following command ```CUDA_VISIBLE_DEVICES=1,2,3
python train_speaker_embeddings.py hparams/train_ecapa_tdnn.yaml --data_parallel_backend --data_parallel_count=3```
to run my experiment, however, it seems like it still accessed the cuda:0 that I am running the other experiment

here is my error message:
```
speechbrain.core - Parameter is not finite: Parameter containing:
tensor([nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan,
        nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan, nan],
       device='cuda:0', requires_grad=True)
speechbrain.core - Parameter is not finite: Parameter containing:
tensor([[-2.6658e-02,  8.7483e-03, -3.4362e-02,  ..., -1.9040e-02,
          2.2166e-02, -3.0037e-02],
        [-6.8921e-04, -7.1734e-05,  1.5596e-04,  ...,  1.7386e-06,
          5.4680e-06, -8.7531e-05],
        [ 7.4638e-04, -5.4538e-05,  1.0013e-05,  ...,  1.7373e-06,
          6.9947e-06, -1.4809e-04],
        ...,
        [-1.0935e-03, -7.2480e-05, -3.2643e-05,  ..., -1.0808e-05,
          6.3613e-05, -1.1533e-03],
        [ 3.3602e-04, -4.2033e-05,  8.1406e-06,  ...,  1.3459e-06,
          2.7885e-06, -4.5925e-06],
        [-1.0819e-06, -5.2945e-07,  1.1754e-07,  ...,  1.6078e-08,
          3.3266e-08, -2.5519e-07]], device='cuda:0', requires_grad=True)
 78%|███████████████████████████████████████████████████████████████████████████▌                     | 30180/38732 [05:25<41:28,  3.44it/s, train_loss=0.198]
speechbrain.core - Exception:
Traceback (most recent call last):
  File "train_speaker_embeddings.py", line 249, in <module>
    speaker_brain.fit(
  File "/mnt/md2/user_winston/speechbrain/speechbrain/core.py", line 1013, in fit
    loss = self.fit_batch(batch)
  File "/mnt/md2/user_winston/speechbrain/speechbrain/core.py", line 842, in fit_batch
    if self.check_gradients(loss):
  File "/mnt/md2/user_winston/speechbrain/speechbrain/core.py", line 875, in check_gradients
    raise ValueError(
ValueError: Loss is not finite and patience is exhausted. To debug, wrap fit() with autograd's `detect_anomaly()`, e.g.

with torch.autograd.detect_anomaly():
        brain.fit(...)
```

Recipe: speechbrain/recipes/VoxCeleb/SpeakerRe
Python: 3.8



Discussion:
Hi @jamfly , thank you for reporting, let me try to reproduce it.
So the GPU used is 0,1,2 instead of 1,2,3 right ?
Hi @aheba, GPU used is 0,1,2,3 (all of my GPUs), instead of 1,2,3
Great, let me do a fix

Resolving patch:
diff --git a/speechbrain/core.py b/speechbrain/core.py
--- a/speechbrain/core.py
+++ b/speechbrain/core.py
@@ -193,12 +193,6 @@ def parse_arguments(arg_list):
         default="cuda:0",
         help="The device to run the experiment on (e.g. 'cuda:0')",
     )
-    parser.add_argument(
-        "--data_parallel_count",
-        type=int,
-        default=-1,
-        help="Number of devices that are used for data_parallel computation",
-    )
     parser.add_argument(
         "--data_parallel_backend",
         default=False,
@@ -218,6 +212,12 @@ def parse_arguments(arg_list):
         default="nccl",
         help="One of {nccl, gloo, mpi}",
     )
+    parser.add_argument(
+        "--find_unused_parameters",
+        default=False,
+        action="store_true",
+        help="This flag disable unused parameters detection",
+    )
     parser.add_argument(
         "--jit_module_keys",
         type=str,
@@ -267,17 +267,8 @@ def parse_arguments(arg_list):
 
     # Checking that DataParallel use the right number of GPU
     if run_opts["data_parallel_backend"]:
-        if run_opts["data_parallel_count"] == 0:
-            raise ValueError(
-                "data_parallel_count must be > 1."
-                "if data_parallel_count = -1, then use all gpus."
-            )
-        if run_opts["data_parallel_count"] > torch.cuda.device_count():
-            raise ValueError(
-                "data_parallel_count must be <= "
-                + str(torch.cuda.device_count())
-                + "if data_parallel_count = -1, then use all gpus."
-            )
+        if torch.cuda.device_count() == 0:
+            raise ValueError("You must have at least 1 GPU.")
 
     # For DDP, the device args must equal to local_rank used by
     # torch.distributed.launch. If run_opts["local_rank"] exists,
@@ -428,10 +419,10 @@ def __init__(  # noqa: C901
             "debug_batches": 2,
             "debug_epochs": 2,
             "device": "cpu",
-            "data_parallel_count": -1,
             "data_parallel_backend": False,
             "distributed_launch": False,
             "distributed_backend": "nccl",
+            "find_unused_parameters": False,
             "jit_module_keys": None,
             "auto_mix_prec": False,
             "max_grad_norm": 5.0,
@@ -461,7 +452,7 @@ def __init__(  # noqa: C901
             sys.exit(
                 "To use data_parallel backend, start your script with:\n\t"
                 "python experiment.py hyperparams.yaml "
-                "--data_parallel_backend=True --data_parallel_count=2"
+                "--data_parallel_backend=True"
                 "To use DDP backend, start your script with:\n\t"
                 "python -m torch.distributed.lunch [args]\n"
                 "experiment.py hyperparams.yaml --distributed_launch=True "
@@ -1094,23 +1085,18 @@ def _wrap_distributed(self):
         elif self.distributed_launch:
             for name, module in self.modules.items():
                 if any(p.requires_grad for p in module.parameters()):
-                    # for ddp, all module must run on same GPU
                     module = SyncBatchNorm.convert_sync_batchnorm(module)
-                    module = DDP(module, device_ids=[self.device])
+                    module = DDP(
+                        module,
+                        device_ids=[self.device],
+                        find_unused_parameters=self.find_unused_parameters,
+                    )
                     self.modules[name] = module
         else:
             # data_parallel_backend
             for name, module in self.modules.items():
                 if any(p.requires_grad for p in module.parameters()):
-                    # if distributed_count = -1 then use all gpus
-                    # otherwise, specify the set of gpu to use
-                    if self.data_parallel_count == -1:
-                        module = DP(module)
-                    else:
-                        module = DP(
-                            module,
-                            [i for i in range(self.data_parallel_count)],
-                        )
+                    module = DP(module)
                     self.modules[name] = module
 
     def evaluate(


Test patch: