Repository: huggingface/accelerate
Pull request: Improve num_processes question in CLI
# Improve the phrasing of asking the number of processes to be used during training

## What does this add?

This slightly tweaks the question asking how many devices the user will be using when running `accelerate config`

## Who is it for?

Closes https://github.com/huggingface/accelerate/issues/338

## Why is it needed?

Stating the number of processes is very general, and I as well found it a bit confusing until I learned that just means number of GPUs, CPUs, etc that will be the **main training device**. As a result, PR proposes changing it from:

```
How many processes in total will you use? [1]
```
To:
```
How many devices should be used for (potentially) distributed training? [1:]
```

Problem statement:
Confusing prompt on accelerator config
When setting up accelerator with deepspeed, the prompt of `how many processes will be used` is really confusing, especially when one only starts with accelerator.

```
In which compute environment are you running? ([0] This machine, [1] AWS (Amazon SageMaker)): 0
Which type of machine are you using? ([0] No distributed training, [1] multi-CPU, [2] multi-GPU, [3] TPU): 2
How many different machines will you use (use more than 1 for multi-node training)? [1]: 1
Do you want to use DeepSpeed? [yes/NO]: yes
What should be your DeepSpeed's ZeRO optimization stage (0, 1, 2, 3)? [2]: 2
Where to offload optimizer states? [NONE/cpu/nvme]: cpu
How many gradient accumulation steps you're passing in your script? [1]: 16
How many processes in total will you use? [1]: 16           <============== this one right here
Do you wish to use FP16 or BF16 (mixed precision)? [NO/fp16/bf16]: fp16
```

I'd suggest maybe something like - `How many GPUs are you using?`

That's assuming I got this line correctly :-)

Hints from issue discussion:
It could be TPUs, or even CPUs if you use distributing training in CPU, so we can't replace it with "How many GPUs". Maybe "devices" ?
Across how many devices should everything be distributed?

Commit message: Rephrase num_processes question

Patch to review:
diff --git a/src/accelerate/commands/config/cluster.py b/src/accelerate/commands/config/cluster.py
index cfab0f7f869..e907effb970 100644
--- a/src/accelerate/commands/config/cluster.py
+++ b/src/accelerate/commands/config/cluster.py
@@ -133,7 +133,7 @@ def get_cluster_input():
         main_training_function = "main"
 
     num_processes = _ask_field(
-        "How many processes in total will you use? [1]: ",
+        "How many devices should be used for (potentially) distributed training? [1:]",
         lambda x: int(x),
         default=1,
         error_message="Please enter an integer.",


Code review comments:
diff_hunk: @@ -133,7 +133,7 @@ def get_cluster_input():
         main_training_function = "main"
 
     num_processes = _ask_field(
-        "How many processes in total will you use? [1]: ",
+        "How many devices should be used for (potentially) distributed training? [1:]",
path: src/accelerate/commands/config/cluster.py
text: @user1:
```suggestion
        "How many devices should be used for (potentially) distributed training? [1]:",
```