Repository: dask/distributed

Issue:
LocalCluster may overallocate memory
On a VM with 16GB of total memory:
- `Client(n_workers=1)` allocates 16GB for a single spawned worker
- `Client(n_workers=2)` allocates 8GB for each of the two spawned workers
- `Client(n_workers=4)` allocates 4GB for each of the four spawned workers
So far so good but
- `Client(n_workers=3)` allocates 8GB for each of the three spawned workers

Even worse, in the latter case there's no `MemoryError` or `KilledWorker` when the workers do end up needing more than the total memory. Instead the whole VM becomes unresponsive and has to be rebooted (couldn't even ssh to it).



Discussion:
Thanks for the bug report @gsakkis I agree that we could probably do better here.

I believe that we're in a case where two reasonable-sounding heuristics are conflicting with each other.

One is here, where we figure out how many threads we should give to each worker given the number of desired processes:

https://github.com/dask/distributed/blob/78a1a347b84ce6835e5b49e4b0a83236cf35a20a/distributed/deploy/local.py#L110-L112

The other is here, where we have workers select an amount of memory proportional to the number of threads that they have received:

https://github.com/dask/distributed/blob/78a1a347b84ce6835e5b49e4b0a83236cf35a20a/distributed/worker.py#L2630-L2634

The simplest solution is, I think, to modify the first heuristic to replace `ceil` with `floor`.  The negative consequence of this choice is that we would undersaturate our CPU if the user selected a number of workers that didn't evenly device the number of logical cores.  This is a less-bad fate than running out of memory though, as you've experienced.

Any interest in making this change?
I'm kinda confused about the logic here; shouldn't the memory per process worker be just `TOTAL_MEMORY / num_processes`? Or `TOTAL_MEMORY / num_threads`  if it needs to be allocated per thread (by the way I thought `threads_per_worker` was applicable only if you instantiate a LocalCluster with `processes=False`, not always) ? In that case the first heuristic could stay as is, e.g. 2 threads per worker for `n_workers=3` = 6 threads in total, with each thread taking 1/6th of `TOTAL_MEMORY`.
LocalCluster doesn't currently specify memory_limit (though it could) it
only specifies threads, which it oversubscribes.  If not given an explicit
memory limit then dask workers look at the total memory and multiply that
by the number of threads they've been told to start divided by the number
of logical cores.

On Tue, Feb 19, 2019 at 12:20 AM George Sakkis <notifications@github.com>
wrote:

> I'm kinda confused about the logic here; shouldn't the memory per process
> worker be just TOTAL_MEMORY / num_processes? Or TOTAL_MEMORY / num_threads
> if it needs to be allocated per thread (by the way I thought
> threads_per_worker was applicable only if you instantiate a LocalCluster
> with processes=False, not always) ? In that case the first heuristic
> could stay as is, e.g. 2 threads per worker for n_workers=3 = 6 threads
> in total, with each thread taking 1/6th of TOTAL_MEMORY.
>
> —
> You are receiving this because you commented.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/2536#issuecomment-465032302>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/AASszPyN4bsvwuwt1ulKeQn1FLZX1UZOks5vO7OwgaJpZM4bBuJe>
> .
>


Resolving patch:
diff --git a/distributed/deploy/local.py b/distributed/deploy/local.py
--- a/distributed/deploy/local.py
+++ b/distributed/deploy/local.py
@@ -18,7 +18,7 @@
         log_errors, thread_state, parse_timedelta)
 from ..nanny import Nanny
 from ..scheduler import Scheduler
-from ..worker import Worker, _ncores
+from ..worker import Worker, parse_memory_limit, _ncores
 
 logger = logging.getLogger(__name__)
 
@@ -110,6 +110,8 @@ def __init__(self, n_workers=None, threads_per_worker=None, processes=True,
         if n_workers and threads_per_worker is None:
             # Overcommit threads per worker, rather than undercommit
             threads_per_worker = max(1, int(math.ceil(_ncores / n_workers)))
+        if n_workers and 'memory_limit' not in worker_kwargs:
+            worker_kwargs['memory_limit'] = parse_memory_limit('auto', 1, n_workers)
 
         worker_kwargs.update({
             'ncores': threads_per_worker,
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -2627,11 +2627,11 @@ class Reschedule(Exception):
     pass
 
 
-def parse_memory_limit(memory_limit, ncores):
+def parse_memory_limit(memory_limit, ncores, total_cores=_ncores):
     if memory_limit is None:
         return None
     if memory_limit == 'auto':
-        memory_limit = int(TOTAL_MEMORY * min(1, ncores / _ncores))
+        memory_limit = int(TOTAL_MEMORY * min(1, ncores / total_cores))
     with ignoring(ValueError, TypeError):
         x = float(memory_limit)
         if isinstance(x, float) and x <= 1:


Test patch:
diff --git a/distributed/deploy/tests/test_local.py b/distributed/deploy/tests/test_local.py
--- a/distributed/deploy/tests/test_local.py
+++ b/distributed/deploy/tests/test_local.py
@@ -328,15 +328,17 @@ def test_remote_access(loop):
         sync(loop, assert_can_connect_from_everywhere_4_6, c.scheduler.port)
 
 
-def test_memory(loop):
-    with LocalCluster(scheduler_port=0, processes=False, silence_logs=False,
-                      diagnostics_port=None, loop=loop) as cluster:
+@pytest.mark.parametrize('n_workers', [None, 3])
+def test_memory(loop, n_workers):
+    with LocalCluster(n_workers=n_workers, scheduler_port=0, processes=False,
+                      silence_logs=False, diagnostics_port=None, loop=loop) as cluster:
         assert sum(w.memory_limit for w in cluster.workers) <= TOTAL_MEMORY
 
 
-def test_memory_nanny(loop):
-    with LocalCluster(scheduler_port=0, processes=True, silence_logs=False,
-                      diagnostics_port=None, loop=loop) as cluster:
+@pytest.mark.parametrize('n_workers', [None, 3])
+def test_memory_nanny(loop, n_workers):
+    with LocalCluster(n_workers=n_workers, scheduler_port=0, processes=True,
+                      silence_logs=False, diagnostics_port=None, loop=loop) as cluster:
         with Client(cluster.scheduler_address, loop=loop) as c:
             info = c.scheduler_info()
             assert (sum(w['memory_limit'] for w in info['workers'].values())