Repository: dask/distributed

Issue:
Gathers & memory issues
I'm opening an anecdotal reporting thread that may hopefully inform usability improvements and bug fixes if/when found. I hope to get more specific as I uncover more & will continue to add info.

In data-heavy dynamic task-of-task workflows, where tasks may return pandas DataFrames on the order of a few GB, users will sometimes create task graphs where one worker client task tries to gather several tasks from many other worker clients (having yielded from the thread pool to gather), but where the total data footprint of the resulting gather will be well into the tens of GBs. Including overhead, this can overwhelm the receiving worker, and start cascading failures because of memory exhaustion.

Ignoring for the moment any design flaws in the program formulation, there are some strange observable behaviors worth reporting:

1 - the nicest and most clear thing that happens is a KilledWorker error, identifying the task that OOM'd the worker N times (for N = allowed-failures+1)

2 - using a LocalCluster w/o a memory limit, I think the nanny dies too because it's co-located within the worker process when the OS (Linux in my case) kills the proc? Anecdotally, what happens SOMETIMES is that tasks are seen to be cancelled, with none found in an error state. Almost always some sort of FATAL-level logging regarding key not found errors are reported by distributed.

3 - in the worst case, a gather thinks it's actually *succeeded*, but at least one of the gathered results is garbled: it comes back either a string instead of a dataframe, or a malformed dataframe.  To see what was happening, I set up one of these degenerate programs to pickle the resulting dataframe from gather to disk when I identified it as garbled (I did this by checking a property I know to be True, specifically if the index is_monotonic_increasing, which showed up as False). Trying to read back the dumped pickle, however, leads to "UnpicklingError: pickle exhausted before end of frame", further indicating data corruption. This is quite hard to reproduce even in my "real" example.

I have yet to produce an example I can share, but I wanted to start a conversation around the observations.




Discussion:
Here's a more detailed trace of one weirdly behaved run. I wrote a scheduler plugin to log when workers leave (which in my case, are because they are memory-killed), and also when tasks transition to an 'erred' state. 

The following comes from a local cluster with six cores and allowed failures = 2. 

My versions btw are distributed 1.21.8 / dask 0.17.4

```
... first, several workers will be killed, having run out of memory. (the number prior to the log level, btw, is the process pid)

2018-06-04 17:08:00,806 17567 [FATAL] distributed.scheduler scheduler.py:1864 | Tried writing to closed comm: {'key': '<task id 1>', 'op': 'lost-data'}
2018-06-04 17:08:00,810 17567 [FATAL] distributed.scheduler scheduler.py:1864 | Tried writing to closed comm: {'key': '<task id 2>', 'op': 'lost-data'}
2018-06-04 17:08:00,812 17567 [FATAL] distributed.scheduler scheduler.py:1864 | Tried writing to closed comm: {'key': '<task id 2>', 'op': 'cancelled-key'}
2018-06-04 17:08:00,813 17567 [FATAL] distributed.scheduler scheduler.py:1864 | Tried writing to closed comm: {'key': '<task id 3>', 'op': 'lost-data'}
2018-06-04 17:08:00,815 17567 [FATAL] distributed.scheduler scheduler.py:1864 | Tried writing to closed comm: {'key': '<task id 4>', 'op': 'lost-data'}
2018-06-04 17:08:00,817 17567 [FATAL] distributed.scheduler scheduler.py:1864 | Tried writing to closed comm: {'key': '<task id 4>', 'op': 'cancelled-key'}
2018-06-04 17:08:00,819 17567 [FATAL] distributed.scheduler scheduler.py:1864 | Tried writing to closed comm: {'key': '<task id 5>', 'op': 'lost-data'}
2018-06-04 17:08:00,826 17567 [FATAL] distributed.scheduler scheduler.py:1864 | Tried writing to closed comm: {'key': '<task id 6>', 'op': 'lost-data'}
2018-06-04 17:08:00,828 17567 [FATAL] distributed.scheduler scheduler.py:1864 | Tried writing to closed comm: {'key': '<task id 6>', 'op': 'cancelled-key'}
2018-06-04 17:08:00,831 17567 [FATAL] distributed.scheduler scheduler.py:1864 | Tried writing to closed comm: {'key': '<task id 7>', 'op': 'lost-data'}
2018-06-04 17:08:00.848 17567 [WARNING] my_plugin:703 | [SCHEDULER] Worker tcp://x.x.x.x:36932 was removed from cluster

... this repeats several times as workers are killed ... 

... finally, i see a fatal log message saying showing a write task-erred op failed ...  but nothing is transitioning to 'erred' state in my plugin!! so my user code never sees this error...
	
2018-06-04 17:11:11,247 17567 [FATAL] distributed.scheduler scheduler.py:1864 | Tried writing to closed comm: {'traceback': None, 'key': '<task id 8>', 'exception': b'\x80\x03cdistributed.scheduler\nKilledWorker...', 'op': 'task-erred'}
2018-06-04 17:11:11,248 17567 [FATAL] distributed.scheduler scheduler.py:1864 | Tried writing to closed comm: {'traceback': None, 'key': '<task id 9>', 'exception': b'\x80\x03cdistributed.scheduler\nKilledWorker...', 'op': 'task-erred'}
2018-06-04 17:11:11.254 17567 [WARNING] my_plugin:703 | [SCHEDULER] Worker tcp://x.x.x.x:36760 was removed from cluster

... things keep chugging along. finally (about 20s later in this case), one of my task results is observed as corrupted - the following error is raised in my code. the dask/distributed error above went unobserved. btw, i haven't inspected the corrupted value yet ... but I would bet on it being a task id instead of a task result. this error also seems to get swallowed. EDIT: yes, it's a task id.
	
2018-06-04 17:11:33.880 17567 [WARNING] my_plugin:699 | [SCHEDULER] Error #1 observed in task <task id 10> on tcp://x.x.x.x:55579: Logic error: result of type <class 'str'> is not DataFrame

... now, another 50 seconds or so later in some worker, i observe that I haven't gotten all my gathered tasks as expected. this is a gather with skipping errors.  I then check for any tasks where task.status == 'error', (which i should have observed much earlier). and yet I still find none in an error state (all ungathered tasks in cancelled state??!!)

2018-06-04 17:12:21.191 18846 [ERROR] my_business_logic:147 | <<< [FAILURE] gathered 10 results < 14 expected in <task id 11>
2018-06-04 17:12:21.192 18846 [ERROR] my_business_logic:195 | ---- NO TASK ERROR(S)s FOUND
```



Ok, I'm getting closer to a reproducible example (versions 1.21.8 / 0.17.4).  My first attempt to demonstrate weird behavior is the following.  Note, my dask config yml has the following:

```
# Scheduler options
work-stealing: True
allowed-failures: 2

# worker options
multiprocessing-method: spawn
worker-memory-pause: False
```

Here is the (first) attempted repro. (Using 6 cores).

```
from distributed import LocalCluster, Client, worker_client, as_completed, get_worker

import numpy as np
import pandas as pd
import time

N = 500   # change me

def big_df(nrows):
    print("in big-df {}".format(get_worker().get_current_task()))
    df = pd.DataFrame(np.ones([nrows, 10000]))
    print("returning {}".format(nrows))
    return df


def task2(sz):
    with worker_client() as wc:
        print("in task two {}".format(get_worker().get_current_task()))
        f1 = wc.submit(big_df, sz)
        f2 = wc.submit(lambda x: x + 1, f1)

        y = f2.result()

        print("Task2 - type {}".format(type(y)))

        return y

def task1():
    with worker_client() as wc:
        print("in task one {}".format(get_worker().get_current_task()))
        futs = []

        for sz in [N] * 6:
            f1 = wc.submit(task2, sz)
            f2 = wc.submit(lambda x: x + 1, f1)
            f3 = wc.submit(lambda x: x + 2, f2)
            futs.append(f3)

        y = wc.gather(futs)

        for df in y:
            print("Task 1 - type {}".format(type(df)))

        return pd.concat(y)


def test():
    lc = LocalCluster(n_workers=6, silence_logs=True, memory_limit=1 << 30)
    c = Client(lc)
    print("in client")
    futs = []

    for i in range(3):
        futs.append(c.submit(task1))

    y = c.gather(futs)
    for df in y:
        print("Client - type {}".format(type(df)))

    c.close()
    lc.close()


if __name__ == '__main__':
    test()
```

If I run with N=500 as the row size, I get:

```
in client
in task one task1-f717f889f888f550c99b4bf36d0ce5d6
in task two task2-bf1bf89b487b14d8eceb54fd68c7565e
in big-df big_df-9bcfb25882a8c7d96269728328d72c60
returning 500
Task2 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Client - type <class 'pandas.core.frame.DataFrame'>
Client - type <class 'pandas.core.frame.DataFrame'>
Client - type <class 'pandas.core.frame.DataFrame'>
```

Good!

But if I run it again, the hash of all the tasks change (why?)

Now, if I run with N = 1000 as the row size, inducing killed workers, the hashes of big_df and task2 start to change mid-program. The KilledWorker failure does not propagate up to task1, so it is run again and again (presumably, it ought to fail immediately after the third time).

```
in client
in task one task1-c2da8337b04ddcf9643ac4c8e622d758
in task two task2-b2c0c31b5016aa73ef28fe7568f0e5de
in big-df big_df-74b485d7b0e044724281da958ad6bd50
returning 1000
Task2 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
in task one task1-c2da8337b04ddcf9643ac4c8e622d758
in task two task2-1f9ffd23b6bad9bc5816900847d62f77
in big-df big_df-6586f3f4fde866763a088c0136634b24
returning 1000
Task2 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
Task 1 - type <class 'pandas.core.frame.DataFrame'>
... etc ...
^C
```

I can't get to repro of bad / garbled results yet, but I'm working on it!





Interesting. In my example above when N = 1000, it looks like the worker dies after its tasks have all transitioned to memory, while sending the results, so no task is in a running state to be marked as a failure when the worker is removed. Ie, the failure is happening after the task1 has committed the concat to memory, and while it's transmitting its result to the client.  If I change `[N] * 6` to `[N] * 12`, I get the expected KilledWorker (b/c it fails on the concat line, not after the task is done).
(Lesson being, workers can be killed as a result of *transmitting* their task results, not just while *running* their tasks.)
Got a repro of problem (3)!  Leads to:

```
...
Task 2 - type <class 'str'>
Task 2 - type <class 'pandas.core.frame.DataFrame'>
Task 2 - type <class 'pandas.core.frame.DataFrame'>
Task 2 - type <class 'pandas.core.frame.DataFrame'>
task 2 done concat
Traceback (most recent call last):
  File "testme.py", line 85, in <module>
    test()
  File "testme.py", line 76, in test
    y = c.gather(futs)
  File " distributed/distributed/client.py", line 1508, in gather
    asynchronous=asynchronous)
  File " distributed/distributed/client.py", line 615, in sync
    return sync(self.loop, func, *args, **kwargs)
  File " distributed/distributed/utils.py", line 253, in sync
    six.reraise(*error[0])
  File " lib/python3.5/site-packages/six.py", line 686, in reraise
    raise value
  File " distributed/distributed/utils.py", line 238, in f
    result[0] = yield make_coro()
  File " lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File " lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File " lib/python3.5/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File " distributed/distributed/client.py", line 1385, in _gather
    traceback)
  File " lib/python3.5/site-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "testme.py", line 56, in task1
    y = wc.gather(futs)
  File " distributed/distributed/client.py", line 1508, in gather
    asynchronous=asynchronous)
  File " distributed/distributed/client.py", line 615, in sync
    return sync(self.loop, func, *args, **kwargs)
  File " distributed/distributed/utils.py", line 253, in sync
    six.reraise(*error[0])
  File " lib/python3.5/site-packages/six.py", line 686, in reraise
    raise value
  File " distributed/distributed/utils.py", line 238, in f
    result[0] = yield make_coro()
  File " lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File " lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File " lib/python3.5/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File " distributed/distributed/client.py", line 1385, in _gather
    traceback)
  File " lib/python3.5/site-packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "testme.py", line 41, in task2
    return pd.concat(y)
  File " lib/python3.5/site-packages/pandas/core/reshape/concat.py", line 212, in concat
    copy=copy)
  File " lib/python3.5/site-packages/pandas/core/reshape/concat.py", line 272, in __init__
    raise TypeError(msg)
TypeError: cannot concatenate object of type "<class 'str'>"; only pd.Series, pd.DataFrame, and pd.Panel (deprecated) objs are valid
```

Code that led to above:

```
from distributed import LocalCluster, Client, worker_client, as_completed, get_worker

from distributed.diagnostics.plugin import SchedulerPlugin

import numpy as np
import pandas as pd
import time

def big_df(nrows):
    print("in big-df {}".format(get_worker().get_current_task()))
    df = pd.DataFrame(np.ones([nrows, 10000]))
    print("returning {}".format(nrows))
    return df


def task2(s):
    with worker_client() as wc:
        print("in task 2 {}".format(get_worker().get_current_task()))

        futs = []

        for sz in [200 * s, 400 * s, 600 * s, 800 * s]:
            f1 = wc.submit(big_df, sz)
            f2 = wc.submit(lambda x: x + 1, f1)
            f3 = wc.submit(lambda x: x + 2, f2)
            futs.append(f3)

        y = wc.gather(futs)

        for df in y:
            print("Task 2 - type {}".format(type(df)))

        try:
            return pd.concat(y)
        finally:
            print("task 2 done concat")

def task1():
    with worker_client() as wc:
        print("in task 1 {}".format(get_worker().get_current_task()))
        futs = []

        for sz in range(1, 3):
            f1 = wc.submit(task2, sz)
            f2 = wc.submit(lambda x: x + 1, f1)
            f3 = wc.submit(lambda x: x + 2, f2)
            futs.append(f3)

        y = wc.gather(futs)

        for df in y:
            print("Task 1 - type {}".format(type(df)))

        try:
            return pd.concat(y)
        finally:
            print("task 1 done concat")


def test():
    lc = LocalCluster(n_workers=6, silence_logs=True, memory_limit=1 << 30)
    c = Client(lc)
    print("in client")
    futs = []

    for i in range(3):
        futs.append(c.submit(task1))

    y = c.gather(futs)
    for df in y:
        print("Client - type {}".format(type(df)))

    c.close()
    lc.close()


if __name__ == '__main__':
    test()
```

Thanks for the effort in creating an example here @adamklein .  Can I interest you in trying out https://github.com/dask/distributed/pull/2037 and seeing if that changes the exception you receive?
Just tested this change in isolation on top of 1.21.8 ... doesn't seem to help.

If a worker is killed while a client (worker or scheduler) is gathering results from that worker, perhaps it would make sense to also mark the tasks in memory and being gathered from that worker as suspicious (and not just the tasks that are processing)?
Ok, anecdotally, I'm seeing an eviction of a key right on the worker (via self.data.fast.evict) right before the gather result retrieved for that key is the key name instead of the value.  Can it be that it is in the middle of sending the result which is evicted? 

EDIT: yes, always see evictions occuring, but doesn't necessarily have to involve the key whose result is garbled...
Ok, after a lot of line debugging, I seem to have figured out the issue.  The eviction is a red herring.

What I see is that in the `while True:` loop of `client._gather`, the `data = {}` is defined locally within the while block, but the `keys` list is nonlocal. The `if local_worker` fills `data` with a key if it is found local to the worker, and also overwrites `keys` as follows:

    keys = [k for k in keys if k not in data]

The problem is, if for some reason the gather response != 'OK' for the remote data in the next block (getting remote data), it loops the while clause again, resetting data to be empty, but having dropped the key, so it is never retrieved from the local worker.  One suggested fix is to move:

    futures2, keys = unpack_remotedata(futures, byte_keys=True)
    keys = [tokey(key) for key in keys]
    bad_data = dict()

within the while loop.

This appears to have the desired effect:

```
diff --git a/distributed/client.py b/distributed/client.py
index ff1d96e3..72e0d38f 100644
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -1336,7 +1336,7 @@ class Client(Node):
     @gen.coroutine
     def _gather(self, futures, errors='raise', direct=None, local_worker=None):
         futures2, keys = unpack_remotedata(futures, byte_keys=True)
-        keys = [tokey(key) for key in keys]
+        orig_keys = [tokey(key) for key in keys]
         bad_data = dict()

         if direct is None:
@@ -1357,6 +1357,8 @@ class Client(Node):
                 raise AllExit()

         while True:
+            keys = orig_keys[:]
+
             logger.debug("Waiting on futures to clear before gather")

             with ignoring(AllExit):
```
Err, maybe it would make more sense for `data = {}` to be defined alongside `bad_data = dict()`.  I think that's even better. Not sure why we'd throw out results that were properly gathered the first time, seems wasteful.
Thanks for doing all of the work to track this down and suggest a solution @adamklein !  What you're saying here makes sense to me.  I really appreciate all of the effort here.
@adamklein is this something that you would be able to submit as a pull request?  Also, do you have any thoughts on how to test this (this isn't strictly necessary, but would be useful to ensure that this kind of thing doesn't recur)?

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -1461,7 +1461,7 @@ def wait(k):
                     else:
                         raise ValueError("Bad value, `errors=%s`" % errors)
 
-            keys = [k for k in keys if k not in bad_keys]
+            keys = [k for k in keys if k not in bad_keys and k not in data]
 
             if local_worker:  # look inside local worker
                 data.update({k: local_worker.data[k]


Test patch: