Repository: huggingface/accelerate

Issue:
Multi machine training not working
I am trying to run my training code on 2 machines. Each of them has 2 GPUs. However, it seems the program runs separately and do not fasten the training progress. Here is my `config.yaml`

machine 1:

```
compute_environment: LOCAL_MACHINE
distributed_type: MULTI_GPU
fp16: true
machine_rank: 0
main_process_ip: 192.168.0.1
main_process_port: 99999
main_training_function: main
num_machines: 2
num_processes: 4

```

machine 2:

```
compute_environment: LOCAL_MACHINE
distributed_type: MULTI_GPU
fp16: true
machine_rank: 1
main_process_ip: 192.168.0.1
main_process_port: 99999
main_training_function: main
num_machines: 2
num_processes: 4

```


Discussion:
The two configs look right to me. Are you launching your script with `accelerate launch` afterward on both machines? What do you mean the programs run separately and do not fasten the training progress?
> The two configs look right to me. Are you launching your script with `accelerate launch` afterward on both machines? What do you mean the programs run separately and do not fasten the training progress?

I run `accelerate launch` in both two machines and the program runs as same as the single machines. The number of batches and total duration of each epoch do not change. Will be `accelerator` wait until all machines start?
Normally you shouldn't see anything until all machines have started, yes.

Resolving patch:
diff --git a/src/accelerate/commands/launch.py b/src/accelerate/commands/launch.py
--- a/src/accelerate/commands/launch.py
+++ b/src/accelerate/commands/launch.py
@@ -66,10 +66,10 @@ def launch_command_parser(subparsers=None):
         "--num_processes", type=int, default=None, help="The total number of processes to be launched in parallel."
     )
     parser.add_argument(
-        "--num_machines", type=int, default=1, help="The total number of machines used in this training."
+        "--num_machines", type=int, default=None, help="The total number of machines used in this training."
     )
     parser.add_argument(
-        "--machine_rank", type=int, default=0, help="The rank of the machine on which this script is launched."
+        "--machine_rank", type=int, default=None, help="The rank of the machine on which this script is launched."
     )
     parser.add_argument("--main_process_ip", type=str, default=None, help="The IP address of the machine of rank 0.")
     parser.add_argument(
@@ -298,6 +298,16 @@ def launch_command(args):
         if not args.multi_gpu and not args.tpu:
             args.multi_gpu = defaults.distributed_type == DistributedType.MULTI_GPU
             args.tpu = defaults.distributed_type == DistributedType.TPU
+        if defaults.compute_environment == ComputeEnvironment.LOCAL_MACHINE:
+            # Update args with the defaults
+            for name, attr in defaults.__dict__.items():
+                # Those args are handled separately
+                if (
+                    name not in ["compute_environment", "fp16", "distributed_type"]
+                    and getattr(args, name, None) is None
+                ):
+                    setattr(args, name, attr)
+
         if args.num_processes is None and defaults.compute_environment == ComputeEnvironment.LOCAL_MACHINE:
             args.num_processes = defaults.num_processes
         if not args.fp16:


Test patch: