Repository: huggingface/accelerate
Pull request: Allow notebook_launcher to launch to multiple GPUs from Colab
# What does this PR do?

Prior behavior for `notebook_launcher` was to ignore the `num_processes` parameter if `in_colab` and simply launch to one CPU/GPU. Since multiple GPUs can be accessed through Colab if the connected VM has multiple GPUs, this behavior was removed. If running in Colab and `num_processes` was unspecified, it will be assumed to be 1 to not break existing code, however if `num_processes` is specified, that value will no longer be ignored and accelerate will launch distributed training.

Fixes #2540 


## Before submitting
- [ ] This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
- [x] Did you read the [contributor guideline](https://github.com/huggingface/accelerate/blob/main/CONTRIBUTING.md#submitting-a-pull-request-pr),
      Pull Request section?
- [x] Was this discussed/approved via a Github issue or the [forum](https://discuss.huggingface.co/)? Please add a link
      to it if that's the case.
- [x] Did you make sure to update the documentation with your changes? Here are the
      [documentation guidelines](https://github.com/huggingface/accelerate/tree/main/docs), and
      [here are tips on formatting docstrings](https://github.com/huggingface/accelerate/tree/main/docs#writing-documentation---specification).
- [ ] Did you write any new necessary tests? _No new test were necessary._


## Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@muellerzr

Problem statement:
Can't utilize multiple GPUs on Colab environment
### System Info

```Shell
- `Accelerate` version: 0.27.2
- Platform: Linux-6.1.58+-x86_64-with-glibc2.35
- Python version: 3.10.12
- Numpy version: 1.23.5
- PyTorch version (GPU?): 2.1.0+cu121 (True)
- PyTorch XPU available: False
- PyTorch NPU available: False
- System RAM: 14.64 GB
- GPU type: Tesla T4
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: no
        - use_cpu: False
        - debug: False
        - num_processes: 4
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - downcast_bf16: no
        - tpu_use_cluster: False
        - tpu_use_sudo: False
        - tpu_env: []
```


### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] One of the scripts in the examples/ folder of Accelerate or an officially supported `no_trainer` script in the `examples` folder of the `transformers` repo (such as `run_no_trainer_glue.py`)
- [X] My own task or dataset (give details below)

### Reproduction

1. Deploy a GCE virtual machine with >1 GPUs.
2. Connect to said VM on Google Colab.
3. Create the accelerate config with the correct number of GPUs.
4. Restart the environment and run the code.

### Expected behavior

It doesn't seem to be possible to get accelerate to use multiple GPUs in a Google Colab environment, even if the VM Colab is connected to has multiple GPUs. The GPUs are available, with `torch.cuda.device_count()` returning the expected number of GPUs.

I believe the issue is in `launchers.py`, where we have the following problematic code:
```
elif in_colab:
    # No need for a distributed launch otherwise as it's either CPU or one GPU.
    if torch.cuda.is_available():
        print("Launching training on one GPU.")
    else:
        print("Launching training on one CPU.")
    function(*args)
```
There is very much a need for distributed launch capabilities, even in a Colab environment. I'm really unsure as to why this code was ever added in the first place... if the config specifies 4 GPUs then accelerate should try to use 4 GPUs, not just ignore this (silently no less, there is no indication of *why* it launches on one GPU so the user needs to delve into the code to figure it out).

Hints from issue discussion:
This is quite new having colab support multiple GPUs, so thanks for letting us know that this is something that's possible now. This was this behavior before because colab did not have this capability. 

Do note that `accelerate config`/`notebook_launcher` are not tied in any way. If you're doing that please just run `!accelerate launch`.

Would you like to open a PR enabling this for users?
Oh, are you saying `notebook_launcher` ignores the accelerate config file? If so I feel like [this documentation on launching from notebook](https://huggingface.co/docs/accelerate/basic_tutorials/notebook#using-the-notebooklauncher) could use a change, because it is pretty confusing for the documentation to state that a config must exist "before any training can be performed", then state that `notebook_launcher` should be used to begin distributed training while making no mention of `accelerate launch`. The associated example notebook is the same.

I'd be happy to work on a PR enabling this for users, but I may need some feedback as I've never contributed to any huggingface open source before. I'll try to get to it later in the week and link to the PR here.
Yes that does indeed need updating. 

Looking forward to the PR!

Commit message: clarified documentation on notebook_launcher (that config file is ignored by notebook_launcher)

Patch to review:
diff --git a/docs/source/basic_tutorials/notebook.md b/docs/source/basic_tutorials/notebook.md
index b1aaab93553..724002e50fb 100644
--- a/docs/source/basic_tutorials/notebook.md
+++ b/docs/source/basic_tutorials/notebook.md
@@ -443,6 +443,12 @@ epoch 4: 94.71
 
 And that's it!
 
+Please note that [`notebook_launcher`] ignores the 🤗 Accelerate config file, to launch based on the config use:
+
+```bash
+accelerate launch
+```
+
 ## Debugging 
 
 A common issue when running the `notebook_launcher` is receiving a CUDA has already been initialized issue. This usually stems
diff --git a/src/accelerate/launchers.py b/src/accelerate/launchers.py
index d92cd7ed553..56ef4d039b4 100644
--- a/src/accelerate/launchers.py
+++ b/src/accelerate/launchers.py
@@ -124,14 +124,14 @@ def train(*args):
         launcher = PrepareForLaunch(function, distributed_type="TPU")
         print(f"Launching a training on {num_processes} TPU cores.")
         xmp.spawn(launcher, args=args, nprocs=num_processes, start_method="fork")
-    elif in_colab:
-        # No need for a distributed launch otherwise as it's either CPU or one GPU.
-        if torch.cuda.is_available():
-            print("Launching training on one GPU.")
-        else:
-            print("Launching training on one CPU.")
-        function(*args)
     else:
+        if in_colab and num_processes is None:
+            # Previous Google Colab behavior assumed only 1 process could be available and ignored the num_processes
+            # parameter. To not break existing notebooks which have omitted this parameter, we can set it to 1 in
+            # this case.
+            num_processes = 1
+            print("Running in Colab and num_processes unspecified. Continuing assuming `num_processes=1`.")
+
         if num_processes is None:
             raise ValueError(
                 "You have to specify the number of GPUs you would like to use, add `num_processes=...` to your call."


Code review comments:
diff_hunk: @@ -124,14 +124,14 @@ def train(*args):
         launcher = PrepareForLaunch(function, distributed_type="TPU")
         print(f"Launching a training on {num_processes} TPU cores.")
         xmp.spawn(launcher, args=args, nprocs=num_processes, start_method="fork")
-    elif in_colab:
-        # No need for a distributed launch otherwise as it's either CPU or one GPU.
-        if torch.cuda.is_available():
-            print("Launching training on one GPU.")
-        else:
-            print("Launching training on one CPU.")
-        function(*args)
     else:
+        if in_colab and num_processes is None:
+            # Previous Google Colab behavior assumed only 1 process could be available and ignored the num_processes
+            # parameter. To not break existing notebooks which have omitted this parameter, we can set it to 1 in
+            # this case.
+            num_processes = 1
+            print("Running in Colab and num_processes unspecified. Continuing assuming `num_processes=1`.")
path: src/accelerate/launchers.py
text: @user1:
Let's change this to keep the `elif`, and simply check if the available GPUs is > 2. Let's not be annoying to our users and print something every time they use this.

Something like:
(`get_gpu_info` comes from `accelerate.utils.env`)
```python
    elif in_colab and get_gpu_info()[1] < 2:
            # No need for a distributed launch otherwise as it's either CPU or one GPU.
            if torch.cuda.is_available():
                print("Launching training on one GPU.")
            else:
                print("Launching training on one CPU.")
            function(*args)
```

@author:
I see, that makes more sense. I've updated the PR now. Please let me know if there are any more issues.