Repository: dask/distributed

Issue:
Prefixing UCX/RMM config variables with `distributed.`
<!-- Please do a quick search of existing issues to make sure that this has not been asked before. -->

As discussed in dask/dask#7579, the configuring variables for UCX/RMM are not contained in the namespace of Distributed, despite only being used there. With that in mind, it would make sense moving forward to prefix these variables with `distributed.`, and maybe have some deprecation period to handle scripts using UCX/RMM with the old variables. In general, we should be able to prefix old variables with a new alias:

```diff
diff --git a/distributed/config.py b/distributed/config.py
index 4b1cfb35..bebc0560 100644
--- a/distributed/config.py
+++ b/distributed/config.py
@@ -48,6 +48,7 @@ aliases = {
     "log-length": "distributed.admin.log-length",
     "log-format": "distributed.admin.log-format",
     "pdb-on-err": "distributed.admin.pdb-on-err",
+    "ucx": "distributed.ucx",
 }

 dask.config.rename(aliases)
```

But I'm not sure if that would cover all the cases where UCX/RMM is configured for Dask-CUDA.

cc @pentschev 



Discussion:
I think that covers all the cases, we will only update how they are used in Dask-CUDA and should be good to go. I would only suggest that we consider whether this should be `distributed.ucx` or `distributed.comm.ucx`, or even a more specific namespace such as `distributed.comm.interface.ucx` (`interface` is obviously a suggestion to a namespace that doesn't exist today, but could be `framework` or some better name if someone can think of one).

@quasiben any thoughts on the above?
I am curious why the current format was chosen. There could be some reasoning behind it which I'm unaware of. Maybe @quasiben knows? 
I think the current format was chosen to reduce the amount of text `DASK_UCX__NVLINK` vs `DASK_DISTRIBUTED_UCX__NVLINK`.  The lines were a little long when we were initially bringing UCX into distributed env vars. However, it's probably more important to be consistent than reduce text.

Nesting ucx in `distributed.comm.ucx` make sense to me. If we adopt other tunable networking libraries `distributed.comm.foo` would also work without having to drop another nested level like `interface`

Resolving patch:
diff --git a/distributed/comm/ucx.py b/distributed/comm/ucx.py
--- a/distributed/comm/ucx.py
+++ b/distributed/comm/ucx.py
@@ -108,7 +108,7 @@ def device_array(n):
                     "In order to send/recv CUDA arrays, Numba or RMM is required"
                 )
 
-    pool_size_str = dask.config.get("rmm.pool-size")
+    pool_size_str = dask.config.get("distributed.rmm.pool-size")
     if pool_size_str is not None:
         pool_size = parse_bytes(pool_size_str)
         rmm.reinitialize(
@@ -121,7 +121,7 @@ def device_array(n):
         ucx_create_endpoint = ucp.create_endpoint
         ucx_create_listener = ucp.create_listener
     else:
-        reuse_endpoints = dask.config.get("ucx.reuse-endpoints")
+        reuse_endpoints = dask.config.get("distributed.comm.ucx.reuse-endpoints")
         if (
             reuse_endpoints is None and ucp.get_ucx_version() >= (1, 11, 0)
         ) or reuse_endpoints is False:
@@ -491,12 +491,12 @@ def _scrub_ucx_config():
     # leave UCX to its default configuration
     if any(
         [
-            dask.config.get("ucx.tcp"),
-            dask.config.get("ucx.nvlink"),
-            dask.config.get("ucx.infiniband"),
+            dask.config.get("distributed.comm.ucx.tcp"),
+            dask.config.get("distributed.comm.ucx.nvlink"),
+            dask.config.get("distributed.comm.ucx.infiniband"),
         ]
     ):
-        if dask.config.get("ucx.rdmacm"):
+        if dask.config.get("distributed.comm.ucx.rdmacm"):
             tls = "tcp" if ucx_110 else "tcp,rdmacm"
             tls_priority = "rdmacm"
         else:
@@ -506,17 +506,22 @@ def _scrub_ucx_config():
         # CUDA COPY can optionally be used with ucx -- we rely on the user
         # to define when messages will include CUDA objects.  Note:
         # defining only the Infiniband flag will not enable cuda_copy
-        if any([dask.config.get("ucx.nvlink"), dask.config.get("ucx.cuda_copy")]):
+        if any(
+            [
+                dask.config.get("distributed.comm.ucx.nvlink"),
+                dask.config.get("distributed.comm.ucx.cuda_copy"),
+            ]
+        ):
             tls = tls + ",cuda_copy"
 
-        if dask.config.get("ucx.infiniband"):
+        if dask.config.get("distributed.comm.ucx.infiniband"):
             tls = "rc," + tls
-        if dask.config.get("ucx.nvlink"):
+        if dask.config.get("distributed.comm.ucx.nvlink"):
             tls = tls + ",cuda_ipc"
 
         options = {"TLS": tls, "SOCKADDR_TLS_PRIORITY": tls_priority}
 
-        net_devices = dask.config.get("ucx.net-devices")
+        net_devices = dask.config.get("distributed.comm.ucx.net-devices")
         if net_devices is not None and net_devices != "":
             options["NET_DEVICES"] = net_devices
 
diff --git a/distributed/config.py b/distributed/config.py
--- a/distributed/config.py
+++ b/distributed/config.py
@@ -48,6 +48,8 @@
     "log-length": "distributed.admin.log-length",
     "log-format": "distributed.admin.log-format",
     "pdb-on-err": "distributed.admin.pdb-on-err",
+    "ucx": "distributed.comm.ucx",
+    "rmm": "distributed.rmm",
 }
 
 dask.config.rename(aliases)


Test patch: