Repository: dask/distributed

Issue:
Non-deterministic "Couldn't gather keys" warning
Related to #1932, in that both of these issues will occur in the same script.

I've got a script that looks something like this:

```python
import itertools

from distributed import Client

def multiply(n):
    if n == 5764:
        raise ValueError('5764 is a bad number!')

    return n * 2

def chunks(iterable, chunk_size):
    iterable = iter(iterable)
    chunk = list(itertools.islice(iterable, chunk_size))
    while chunk:
        yield chunk
        chunk = list(itertools.islice(iterable, chunk_size))

def run():
    client = Client()

    print('Starting.')

    for i, chunk in enumerate(chunks(range(10_000), 5000)):
        print(f'Processing chunk #{i}.')
        futures = [client.submit(multiply, nums) for nums in chunk]

        client.gather(futures, errors='skip')

    print('Done.')

if __name__ == '__main__':
    run()
```

Where `range(10_000)` is actually an iterator reading chunks of lines from a file, and multiply is transforming those chunks into something that is then stored in a database. The `if n == 5764: raise ValueError('5764 is a bad number!')` bit is just so that exactly one error is thrown, to demonstrate that the resulting non-deterministic error message is odd.

Running the above script sometimes results in output like this:

```
Starting.
Processing chunk #0.
Processing chunk #1.
distributed.worker - WARNING -  Compute Failed
Function:  multiply
args:      (5764)
kwargs:    {}
Exception: ValueError('5764 is a bad number!',)

distributed.scheduler - ERROR - error from worker tcp://127.0.0.1:62788: 5764 is a bad number!
distributed.client - WARNING - Couldn't gather keys {'multiply-5fa51d35f2029544cde8a9495f63be5e': [], 'multiply-e34b9cd8354b6b107491affa58dd2662': [], 'multiply-bae01f52f83e12cbaa50d9d9a89e1a7f': [], ... snip hundreds of keys ...}
Done.
```

And sometimes results in output like this:

```
Starting.
Processing chunk #0.
Processing chunk #1.
distributed.worker - WARNING -  Compute Failed
Function:  multiply
args:      (5764)
kwargs:    {}
Exception: ValueError('5764 is a bad number!',)

distributed.scheduler - ERROR - error from worker tcp://127.0.0.1:62730: 5764 is a bad number!
Done.
```

The warning doesn't seem to happen for smaller batch sizes (or it's at least much harder to trigger). I see that there's a call to `self._send_to_scheduler` for the failed keys, so I'm wondering if the listed keys are actually failing, or if they're just getting rescheduled and running normally. If the keys are getting rescheduled and run properly, would it make sense to change the warning message to be something like `Couldn't gather keys, rescheduling: <list of keys>`?


Discussion:
If the keys are getting rescheduled and run properly, would it make sense to change the warning message to be something like `Couldn't gather keys, rescheduling: <list of keys>`?

Yes, that would make sense.  Any interest in submitting a small PR with this change?

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -1415,7 +1415,7 @@ def wait(k):
 
             if response['status'] == 'error':
                 log = logger.warning if errors == 'raise' else logger.debug
-                log("Couldn't gather keys %s", response['keys'])
+                log("Couldn't gather %s keys, rescheduling %s", (len(response['keys']), response['keys']))
                 for key in response['keys']:
                     self._send_to_scheduler({'op': 'report-key',
                                              'key': key})


Test patch: