Repository: dask/distributed

Issue:
Simple local cluster on laptop fails with deactivated GPU
<!-- Please include a self-contained copy-pastable example that generates the issue if possible.

Please be concise with code posted. See guidelines below on how to provide a good bug report:

- Craft Minimal Bug Reports http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports
- Minimal Complete Verifiable Examples https://stackoverflow.com/help/mcve

Bug reports that follow these guidelines are easier to diagnose, and so are often handled much more quickly.
-->

**What happened**:
Thinkpad T470p with dual Intel HD Graphics 630 / Nvidia GeForce 940MX GPU ('optimus' configuration). I use [bumblebee](https://wiki.debian.org/Bumblebee) to switch on/off the Nvidia GPU only when I need it for CUDA computational work to save power. The Nvidia is still visible but inactive when switched off this way.

When I start below code snippet with a deactivated Nvidia, it crashes:

```
$ sudo echo OFF > /proc/acpi/bbswitch
$ ./test_distributed.py 
Traceback (most recent call last):
  File "./test_distributed.py", line 6, in <module>
    client = dask.distributed.Client()
  File "/home/sbauer/.local/lib/python3.7/site-packages/distributed/client.py", line 754, in __init__
    self.start(timeout=timeout)
  File "/home/sbauer/.local/lib/python3.7/site-packages/distributed/client.py", line 967, in start
    sync(self.loop, self._start, **kwargs)
  File "/home/sbauer/.local/lib/python3.7/site-packages/distributed/utils.py", line 354, in sync
    raise exc.with_traceback(tb)
  File "/home/sbauer/.local/lib/python3.7/site-packages/distributed/utils.py", line 337, in f
    result[0] = yield future
  File "/home/sbauer/.local/lib/python3.7/site-packages/tornado/gen.py", line 762, in run
    value = future.result()
  File "/home/sbauer/.local/lib/python3.7/site-packages/distributed/client.py", line 1035, in _start
    **self._startup_kwargs,
  File "/home/sbauer/.local/lib/python3.7/site-packages/distributed/deploy/spec.py", line 401, in _
    await self._start()
  File "/home/sbauer/.local/lib/python3.7/site-packages/distributed/deploy/spec.py", line 311, in _start
    self.scheduler = cls(**self.scheduler_spec.get("options", {}))
  File "/home/sbauer/.local/lib/python3.7/site-packages/distributed/scheduler.py", line 3382, in __init__
    import distributed.dashboard.scheduler
  File "/home/sbauer/.local/lib/python3.7/site-packages/distributed/dashboard/scheduler.py", line 11, in <module>
    from .components.nvml import gpu_doc  # noqa: 1708
  File "/home/sbauer/.local/lib/python3.7/site-packages/distributed/dashboard/components/nvml.py", line 22, in <module>
    NVML_ENABLED = nvml.device_get_count() > 0
  File "/home/sbauer/.local/lib/python3.7/site-packages/distributed/diagnostics/nvml.py", line 31, in device_get_count
    return pynvml.nvmlDeviceGetCount()
  File "/home/sbauer/.local/lib/python3.7/site-packages/pynvml/nvml.py", line 1568, in nvmlDeviceGetCount
    _nvmlCheckReturn(ret)
  File "/home/sbauer/.local/lib/python3.7/site-packages/pynvml/nvml.py", line 743, in _nvmlCheckReturn
    raise NVMLError(ret)
pynvml.nvml.NVMLError_Uninitialized: Uninitialized
```

When I activate the Nvidia GPU, all is fine:
```
$ sudo echo ON > /proc/acpi/bbswitch
$ ./test_distributed.py 
<Client: 'tcp://127.0.0.1:42147' processes=4 threads=8, memory=15.55 GiB>
```

Even specifically setting CUDA_VISIBLE_DEVICES to empty doesn't work (I use this env in Tensorflow to explicitly ignore the GPU):
```
$ CUDA_VISIBLE_DEVICES='' ./test_distributed.py
Traceback (most recent call last):
  File "./test_distributed.py", line 6, in <module>
...
```

**What you expected to happen**:
Local cluster starts regardless of active or deactive GPU.

It would help if there were some possibility to tell *distributed* to ignore any present GPU, but I didn't find anything in the documentation.

**Minimal Complete Verifiable Example**:

```python
#!/usr/bin/env python3

import dask.distributed

if __name__ == "__main__":
    client = dask.distributed.Client()
    print(client)
```

**Anything else we need to know?**:

**Environment**:

- Dask version: dask 2021.6.0 / distributed 2021.6.0
- Python version: 3.7.3
- Operating System: Linux 4.19.0-16-amd64 #1 SMP Debian 4.19.181-1 (2021-03-19) x86_64 GNU/Linux
- Install method (conda, pip, source): pip



Discussion:
Thanks for reporting @SteffenBauer! cc @pentschev @quasiben 

EDIT: cross-referencing https://github.com/dask/distributed/pull/4873 which may be related 
A way to tell any process to ignore NVIDIA GPUs is to specify an empty string to `CUDA_VISIBLE_DEVICES`, prepending it to the process or running `export CUDA_VISIBLE_DEVICES=""` before launching Python. Programatically, you can place the following at the top of your code:

```python
import os
os.environ["CUDA_VISIBLE_DEVICES"] = ""
```

Could you try that @SteffenBauer ? Unfortunately, bumblebee can create a number of issues such as the one you're reporting, so I'm not sure we can resolve that in Distributed in a different way.
> A way to tell any process to ignore NVIDIA GPUs is to specify an empty string to `CUDA_VISIBLE_DEVICES`, prepending it to the process or running `export CUDA_VISIBLE_DEVICES=""` before launching Python. Programatically, you can place the following at the top of your code:
> 
> ```python
> import os
> os.environ["CUDA_VISIBLE_DEVICES"] = ""
> ```
> 
> Could you try that @SteffenBauer ? Unfortunately, bumblebee can create a number of issues such as the one you're reporting, so I'm not sure we can resolve that in Distributed in a different way.

I already tried this by setting CUDA_VISIBLE_DEVICES='' with starting the file as script, as mentioned in my initial post.

To be sure, I set it inside the python script, but I still get the crash. It looks to me that Distributed is not observing CUDA_VISIBLE_DEVICES when it is set to empty, in contrast to Tensorflow.

Both
```
#!/usr/bin/env python3

import os
import dask.distributed

if __name__ == "__main__":
    os.environ["CUDA_VISIBLE_DEVICES"] = ""
    client = dask.distributed.Client()
    print(client)
```
and
```
#!/usr/bin/env python3

import os
import dask.distributed
os.environ["CUDA_VISIBLE_DEVICES"] = ""
if __name__ == "__main__":
    client = dask.distributed.Client()
    print(client)
```
fail with `pynvml.nvml.NVMLError_Uninitialized: Uninitialized`
Ah yes, you're right, NVML doesn't respect `CUDA_VISIBLE_DEVICES`, I had this discussion just last week with @quasiben and missed this now again as well as your initial post, sorry about that.

Unfortunately, I don't think there's another way of disabling GPU visibility in NVML. The only option I can see at this point is that we introduce a variable in Distributed to explicitly disable it, such as `DASK_DISTRIBUTED__NVML=False`. What do you think @quasiben , do you see another alternative here?
> The only option I can see at this point is that we introduce a variable in Distributed to explicitly disable it, such as `DASK_DISTRIBUTED__NVML=False`.

I think this would be an acceptable way to solve this issue. The main case where this issue occurs should be (like in my case) people developing dask code on a laptop with a GPU, which is not used and needed. An according environmental variable could be set permanently in .bashrc, for example.


I don't see an alternative here.  I think it make sense to have an option to not load any GPU even if they are present on the system.  `DASK_DISTRIBUTED__NVML` is fine can you think of a more descriptive name ?
> I don't see an alternative here. I think it make sense to have an option to not load any GPU even if they are present on the system. `DASK_DISTRIBUTED__NVML` is fine can you think of a more descriptive name ?

I checked `distributed.yaml`, according to the configuration schema perhaps `DASK_DISTRIBUTED__NVML__ENABLED`? This could allow further configuration options for nvml in the future, should the need for it arise.
@quasiben could you elaborate on what you mean by "not load any GPU"? Currently, I think the only part that actively attempts to address GPUs is the NVML module -- apart from the UCX comms which will create a CUDA context when `cuda-copy=True`. I don't know what more we could prevent from doing that, for instance anybody could use Numba/CuPy or some other GPU library that would create contexts anyway.
You're correct.  Loading GPUs from Numba/CuPy will also be possible so it's just the NVML module

Resolving patch:
diff --git a/distributed/diagnostics/nvml.py b/distributed/diagnostics/nvml.py
--- a/distributed/diagnostics/nvml.py
+++ b/distributed/diagnostics/nvml.py
@@ -1,5 +1,7 @@
 import os
 
+import dask
+
 try:
     import pynvml
 except ImportError:
@@ -12,6 +14,11 @@
 
 def init_once():
     global nvmlInitialized, nvmlLibraryNotFound, nvmlOwnerPID
+
+    if dask.config.get("distributed.diagnostics.nvml") is False:
+        nvmlInitialized = False
+        return
+
     if pynvml is None or (nvmlInitialized is True and nvmlOwnerPID == os.getpid()):
         return
 


Test patch:
diff --git a/distributed/diagnostics/tests/test_nvml.py b/distributed/diagnostics/tests/test_nvml.py
--- a/distributed/diagnostics/tests/test_nvml.py
+++ b/distributed/diagnostics/tests/test_nvml.py
@@ -4,6 +4,8 @@
 
 pynvml = pytest.importorskip("pynvml")
 
+import dask
+
 from distributed.diagnostics import nvml
 from distributed.utils_test import gen_cluster
 
@@ -19,6 +21,23 @@ def test_one_time():
     assert len(output["name"]) > 0
 
 
+def test_enable_disable_nvml():
+    try:
+        pynvml.nvmlShutdown()
+    except pynvml.NVMLError_Uninitialized:
+        pass
+    else:
+        nvml.nvmlInitialized = False
+
+    with dask.config.set({"distributed.diagnostics.nvml": False}):
+        nvml.init_once()
+        assert nvml.nvmlInitialized is False
+
+    with dask.config.set({"distributed.diagnostics.nvml": True}):
+        nvml.init_once()
+        assert nvml.nvmlInitialized is True
+
+
 def test_1_visible_devices():
     if nvml.device_get_count() < 1:
         pytest.skip("No GPUs available")