internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dask/distributed

Issue:
Failure on a 3-core VM in distributed\deploy\tests\test_local.py
```
________________________________ test_defaults ________________________________
distributed\deploy\tests\test_local.py:93: in test_defaults
    assert sum(w.ncores for w in c.workers) == max(2, _ncores)
E   assert 2 == 3
E    +  where 2 = sum(<generator object test_defaults.<locals>.<genexpr> at 0x00
000000084A6048>)
E    +  and   3 = max(2, 3)
```



Discussion:

Resolving patch:
diff --git a/distributed/deploy/local.py b/distributed/deploy/local.py
--- a/distributed/deploy/local.py
+++ b/distributed/deploy/local.py
@@ -1,6 +1,7 @@
 from __future__ import print_function, division, absolute_import
 
 import logging
+import math
 from threading import Thread
 from time import sleep
 
@@ -75,7 +76,8 @@ def __init__(self, n_workers=None, threads_per_worker=None, nanny=True,
         if n_workers is None and threads_per_worker is not None:
             n_workers = max(1, _ncores // threads_per_worker)
         if n_workers and threads_per_worker is None:
-            threads_per_worker = max(1, _ncores // n_workers)
+            # Overcommit threads per worker, rather than undercommit
+            threads_per_worker = max(1, int(math.ceil(_ncores / n_workers)))
 
         self.loop = loop or IOLoop()
         if not self.loop._running:


Test patch:
diff --git a/distributed/deploy/tests/test_local.py b/distributed/deploy/tests/test_local.py
--- a/distributed/deploy/tests/test_local.py
+++ b/distributed/deploy/tests/test_local.py
@@ -90,7 +90,12 @@ def test_defaults():
 
     with LocalCluster(n_workers=2, scheduler_port=0, silence_logs=False,
             diagnostic_port=None) as c:
-        assert sum(w.ncores for w in c.workers) == max(2, _ncores)
+        if _ncores % 2 == 0:
+            expected_total_threads = max(2, _ncores)
+        else:
+            # n_workers not a divisor of _ncores => threads are overcommitted
+            expected_total_threads = max(2, _ncores + 1)
+        assert sum(w.ncores for w in c.workers) == expected_total_threads
 
     with LocalCluster(threads_per_worker=_ncores * 2, scheduler_port=0,
             silence_logs=False, diagnostic_port=None) as c:
Repository: dask/distributed

Issue:
Update changelog to look like dask/dask
The release notes of dask/distributed look different than those in dask/dask (which is a bit fancier).  It would be nice to align these two.  

@dukebody do you have any interest in repeating your effort in https://github.com/dask/dask/pull/1526

https://github.com/dask/distributed/blob/master/release-notes.md


Discussion:
Hey @mrocklin! Yes I can do it, but I won't be available until a week from now. If this ticket can wait that long (I guess so) please assign it to me. :)
Very happy to wait.  Thanks for taking this on @dukebody !

Resolving patch:
diff --git a/docs/source/conf.py b/docs/source/conf.py
--- a/docs/source/conf.py
+++ b/docs/source/conf.py
@@ -35,6 +35,7 @@
     'sphinx.ext.ifconfig',
     'sphinx.ext.viewcode',
     'sphinx.ext.autosummary',
+    'sphinx.ext.extlinks',
     'numpydoc',
 ]
 
@@ -367,3 +368,10 @@
 
 # If false, no index is generated.
 #epub_use_index = True
+
+# Link to GitHub issues and pull requests using :pr:`1234` and :issue:`1234`
+# syntax
+extlinks = {
+    'issue': ('https://github.com/dask/distributed/issues/%s', 'GH#'),
+    'pr': ('https://github.com/dask/distributed/pull/%s', 'GH#')
+}


Test patch:
Repository: dask/distributed

Issue:
Exception handling
`client.gather()` accepts an argument `errors=`. What values are allowed and what do they do?

Are there other methods to work with a collection of futures as a whole where one or more have thrown an exception?

Is there a way to instruct dask (distributed) to persist/cache the inputs to the failed key in the graph?


Discussion:

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -962,9 +962,22 @@ def gather(self, futures, errors='raise', maxsize=0):
         Accepts a future, nested container of futures, iterator, or queue.
         The return type will match the input type.
 
+        Parametrs
+        ---------
+        futures: Collection of futures
+            This can be a possibly nested collection of Future objects.
+            Collections can be lists, sets, iterators, queues or dictionaries
+        errors: string
+            Either 'raise' or 'skip' if we should raise if a future has erred
+            or skip its inclusion in the output collection
+        maxsize: int
+            If the input is a queue then this produces an output queue with a
+            maximum size.
+
         Returns
         -------
-        Future results
+        results: a collection of the same type as the input, but now with
+        gathered results rather than futures
 
         Examples
         --------


Test patch:
Repository: dask/distributed

Issue:
PEP 479 compatibility
[PEP 479](https://www.python.org/dev/peps/pep-0479/) changes style of exit from a generator. Basically, instead of
```python
raise StopIteration
```
one should
```
return
```
Currently, in Python 3.6 `AsCompleted.batches()` generator causes:
> DeprecationWarning: generator 'AsCompleted.batches' raised StopIteration

In Python 3.7 this will be an error, not a warning.


Discussion:
Are you interested in submitting a fix @slavoutich ?

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -2693,7 +2693,10 @@ def batches(self):
         [6]
         """
         while True:
-            yield self.next_batch(block=True)
+            try:
+                yield self.next_batch(block=True)
+            except StopIteration:
+                return
 
 
 as_completed = AsCompleted


Test patch:
Repository: dask/distributed

Issue:
Configurable timeouts
Currently we have set timeouts at 3s sprinkled throughout the code.  It would be useful to make these configurable through the config.yaml file.

cc @pitrou 


Discussion:

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -673,7 +673,7 @@ def _reconnect(self, timeout=0.1):
                     yield gen.sleep(timeout)
 
     @gen.coroutine
-    def _ensure_connected(self, timeout=5):
+    def _ensure_connected(self, timeout=None):
         if (self.scheduler_comm and not self.scheduler_comm.closed() or
             self._connecting_to_scheduler):
             return
diff --git a/distributed/comm/core.py b/distributed/comm/core.py
--- a/distributed/comm/core.py
+++ b/distributed/comm/core.py
@@ -8,6 +8,7 @@
 
 from tornado import gen
 
+from ..config import config
 from ..metrics import time
 from . import registry
 from .addressing import parse_address
@@ -152,12 +153,15 @@ def connect(self, address, deserialize=True):
 
 
 @gen.coroutine
-def connect(addr, timeout=3, deserialize=True, connection_args=None):
+def connect(addr, timeout=None, deserialize=True, connection_args=None):
     """
     Connect to the given address (a URI such as ``tcp://127.0.0.1:1234``)
     and yield a ``Comm`` object.  If the connection attempt fails, it is
     retried until the *timeout* is expired.
     """
+    if timeout is None:
+        timeout = float(config.get('connect-timeout', 3))  # default 3 s.
+
     scheme, loc = parse_address(addr)
     backend = registry.get_backend(scheme)
     connector = backend.get_connector()
diff --git a/distributed/core.py b/distributed/core.py
--- a/distributed/core.py
+++ b/distributed/core.py
@@ -353,7 +353,7 @@ class rpc(object):
     comms = ()
     address = None
 
-    def __init__(self, arg=None, comm=None, deserialize=True, timeout=3,
+    def __init__(self, arg=None, comm=None, deserialize=True, timeout=None,
                  connection_args=None):
         self.comms = {}
         self.address = coerce_to_address(arg)
@@ -550,7 +550,7 @@ def __call__(self, addr=None, ip=None, port=None):
         return PooledRPCCall(addr, self)
 
     @gen.coroutine
-    def connect(self, addr, timeout=3):
+    def connect(self, addr, timeout=None):
         """
         Get a Comm to the given address.  For internal use.
         """


Test patch:
Repository: dask/distributed

Issue:
Docs of Client.close do not reflect behavior
This is probably related to #1085. The [docs](https://distributed.readthedocs.io/en/latest/api.html#distributed.Client.close) suggest `close` shuts down the cluster, while it actually only closes the client's connection.

Also, do I have to call `close` explicitly? I ran into errors at the end of my program:

~~~
tornado.application - ERROR - Exception in callback <functools.partial object at 0x7f845de73208>
Traceback (most recent call last):
  File "/home/vagrant/venvs/experiment/local/lib/python2.7/site-packages/tornado/ioloop.py", line 604, in _run_callback
    ret = callback()
  File "/home/vagrant/venvs/experiment/local/lib/python2.7/site-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/home/vagrant/venvs/experiment/local/lib/python2.7/site-packages/distributed/client.py", line 769, in _dec_ref
    self._release_key(key)
  File "/home/vagrant/venvs/experiment/local/lib/python2.7/site-packages/distributed/client.py", line 780, in _release_key
    'client': self.id})
  File "/home/vagrant/venvs/experiment/local/lib/python2.7/site-packages/distributed/client.py", line 628, in _send_to_scheduler
    raise Exception("Client not running.  Status: %s" % self.status)
Exception: Client not running.  Status: closing
~~~

When explicitly calling `close` the errors do not occur. Unfortunately, I couldn't come up with a minimal reproducible example that yields these errors.



Discussion:

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -876,10 +876,6 @@ def _handle_error(self, exception=None):
         logger.warning("Scheduler exception:")
         logger.exception(exception)
 
-    def close(self, **kwargs):
-        """ Close this client and its connection to the scheduler """
-        return self.sync(self._close, **kwargs)
-
     @gen.coroutine
     def _close(self, fast=False):
         """ Send close signal and wait until scheduler completes """
@@ -910,13 +906,12 @@ def _close(self, fast=False):
     _shutdown = _close
 
     def close(self, timeout=10):
-        """ Send close signal and wait until scheduler terminates
+        """ Close this client
 
-        This cancels all currently running tasks, clears the state of the
-        scheduler, and shuts down all workers and scheduler.
+        Clients will also close automatically when your Python session ends
 
-        You do not need to call this when you finish your session.  You only
-        need to call this if you want to take down the distributed cluster.
+        If you started a client without arguments like ``Client()`` then this
+        will also close the local cluster that was started at the same time.
 
         See Also
         --------


Test patch:
Repository: dask/distributed

Issue:
Catching ImportError for ssl-less environments?
Hi Matthew, dask team,

I've been digging into dask recently and am super impressed. So, thanks for all the work you guys are putting into these awesome libraries. I was amazed at how easy it was to get setup.

I've been looking at ways to use dask in a project at work where SSL is intentionally disabled due to some federal compliance requirements. In this case, I have no SSL module (despite it being in the standard lib).

I noticed that tornado has a pattern to catch ssl import errors:

```
try:
    import ssl
except ImportError:
    # ssl is not available on Google App Engine.
    ssl = None
```
(https://github.com/tornadoweb/tornado/blob/0e869e3f4735685cb2460aac499b509e6f5cc58a/tornado/tcpserver.py#L32)

In this case, adding such a change would allow for us to use distributed despite not having access to the ssl module. If you think this is a reasonable request, I'd love to make a PR. If not, we can simply fork distributed since it is such a small change, however, I thought it wouldn't hurt to ask in case you find this small change amenable.

Best,
Xander



Discussion:
This seems fine to me.
A pull request would be most welcome

Resolving patch:
diff --git a/distributed/comm/tcp.py b/distributed/comm/tcp.py
--- a/distributed/comm/tcp.py
+++ b/distributed/comm/tcp.py
@@ -3,10 +3,14 @@
 import errno
 import logging
 import socket
-import ssl
 import struct
 import sys
 
+try:
+    import ssl
+except ImportError:
+    ssl = None
+
 import tornado
 from tornado import gen, netutil
 from tornado.iostream import StreamClosedError
diff --git a/distributed/security.py b/distributed/security.py
--- a/distributed/security.py
+++ b/distributed/security.py
@@ -1,6 +1,9 @@
 from __future__ import print_function, division, absolute_import
 
-import ssl
+try:
+    import ssl
+except ImportError:
+    ssl = None
 
 from . import config
 


Test patch:
diff --git a/distributed/tests/test_security.py b/distributed/tests/test_security.py
--- a/distributed/tests/test_security.py
+++ b/distributed/tests/test_security.py
@@ -1,9 +1,13 @@
 from __future__ import print_function, division, absolute_import
 
 from contextlib import contextmanager
-import ssl
 import sys
 
+try:
+    import ssl
+except ImportError:
+    ssl = None
+
 import pytest
 from tornado import gen
 
diff --git a/distributed/utils_test.py b/distributed/utils_test.py
--- a/distributed/utils_test.py
+++ b/distributed/utils_test.py
@@ -14,7 +14,6 @@
 import shutil
 import signal
 import socket
-import ssl
 import subprocess
 import sys
 import tempfile
@@ -25,6 +24,11 @@
 import warnings
 import weakref
 
+try:
+    import ssl
+except ImportError:
+    ssl = None
+
 import psutil
 import pytest
 import six
Repository: dask/distributed

Issue:
Future repr issue in Jupyter Notebook
When trying to inspect a `Future` in the Jupyter Notebook, an issue cropped up trying to represent it. Looks like it assumes a `key` is a `str` when it is in fact a `tuple`. Running `distributed` version 1.20.1.

```python
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/opt/conda3/lib/python3.6/site-packages/IPython/core/formatters.py in __call__(self, obj)
    343             method = get_real_method(obj, self.print_method)
    344             if method is not None:
--> 345                 return method()
    346             return None
    347         else:

/opt/conda3/lib/python3.6/site-packages/distributed/client.py in _repr_html_(self)
    334                 typ = str(self.type)
    335             text += '<font color="gray">type: </font>%s, ' % typ
--> 336         text += '<font color="gray">key: </font>%s' % html_escape(self.key)
    337         return text
    338 

/opt/conda3/lib/python3.6/html/__init__.py in escape(s, quote)
     17     translated.
     18     """
---> 19     s = s.replace("&", "&amp;") # Must be done first!
     20     s = s.replace("<", "&lt;")
     21     s = s.replace(">", "&gt;")

AttributeError: 'tuple' object has no attribute 'replace'

<Future: status: finished, type: Array, key: ('store-rechunk-merge-9d56828f48fdb9af2a81e6b10a09d976', 3, 3, 0)>
```


Discussion:

Resolving patch:
diff --git a/distributed/client.py b/distributed/client.py
--- a/distributed/client.py
+++ b/distributed/client.py
@@ -334,7 +334,7 @@ def _repr_html_(self):
             except AttributeError:
                 typ = str(self.type)
             text += '<font color="gray">type: </font>%s, ' % typ
-        text += '<font color="gray">key: </font>%s' % html_escape(self.key)
+        text += '<font color="gray">key: </font>%s' % html_escape(str(self.key))
         return text
 
     def __await__(self):


Test patch:
diff --git a/distributed/tests/test_client.py b/distributed/tests/test_client.py
--- a/distributed/tests/test_client.py
+++ b/distributed/tests/test_client.py
@@ -271,13 +271,23 @@ def test_persist_retries(c, s, a, b):
 
 @gen_cluster(client=True)
 def test_future_repr(c, s, a, b):
+    x = c.submit(inc, 10)
     for func in [repr, lambda x: x._repr_html_()]:
-        x = c.submit(inc, 10)
         assert str(x.key) in func(x)
         assert str(x.status) in func(x)
         assert str(x.status) in repr(c.futures[x.key])
 
 
+@gen_cluster(client=True)
+def test_future_tuple_repr(c, s, a, b):
+    da = pytest.importorskip('dask.array')
+    y = da.arange(10, chunks=(5,)).persist()
+    f = futures_of(y)[0]
+    for func in [repr, lambda x: x._repr_html_()]:
+        for k in f.key:
+            assert str(k) in func(f)
+
+
 @gen_cluster(client=True)
 def test_Future_exception(c, s, a, b):
     x = c.submit(div, 1, 0)
Repository: dask/distributed

Issue:
Error when given an unhashable function
https://github.com/dask/distributed/blob/b258495e5e7e877170de7e0a3f03657dc8b26199/distributed/worker.py#L669

distributed will throw an error when given an unhashable function.
```
>>> import distributed
>>> client = distributed.Client()
>>> dct = {'a': 1}
>>> getter = dct.get
>>> dsk = {'return': (getter, 'a')}
>>> client.get(dsk, 'return')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/distributed/client.py", line 1975, in get
    resources=resources)
  File "/usr/local/lib/python3.6/site-packages/distributed/client.py", line 1936, in _graph_to_futures
    'tasks': valmap(dumps_task, dsk3),
  File "/usr/local/lib/python3.6/site-packages/toolz/dicttoolz.py", line 84, in valmap
    rv.update(zip(iterkeys(d), map(func, itervalues(d))))
  File "/usr/local/lib/python3.6/site-packages/distributed/worker.py", line 742, in dumps_task
    return {'function': dumps_function(task[0]),
  File "/usr/local/lib/python3.6/site-packages/distributed/worker.py", line 705, in dumps_function
    if func not in cache:
TypeError: unhashable type: 'dict'
```

Functions are not hashable if their closure contains unhashable objects. Any time you have code like `if func in dict` you need to wrap it in a try/except block.

The solution is to wrap line 669 in a try/except block.


Discussion:
Agreed.  Thank you for identifying the issue.  

Can I interest you in submitting a PR with a fix?
@mrocklin unfortunately I can't contribute before acquiring approval from my company. Doing that would require some time, it may be faster for someone else to fix it.

Resolving patch:
diff --git a/distributed/worker.py b/distributed/worker.py
--- a/distributed/worker.py
+++ b/distributed/worker.py
@@ -669,13 +669,15 @@ def execute_task(task):
 
 def dumps_function(func):
     """ Dump a function to bytes, cache functions """
-    if func not in cache:
-        b = pickle.dumps(func)
-        if len(b) < 100000:
-            cache[func] = b
-        else:
-            return b
-    return cache[func]
+    try:
+        result = cache[func]
+    except KeyError:
+        result = pickle.dumps(func)
+        if len(result) < 100000:
+            cache[func] = result
+    except TypeError:
+        result = pickle.dumps(func)
+    return result
 
 
 def dumps_task(task):


Test patch:
diff --git a/distributed/tests/test_client.py b/distributed/tests/test_client.py
--- a/distributed/tests/test_client.py
+++ b/distributed/tests/test_client.py
@@ -5136,5 +5136,13 @@ def test_scatter_direct(s, a, b):
 
     yield c._close()
 
+
+@gen_cluster(client=True)
+def test_unhashable_function(c, s, a, b):
+    d = {'a': 1}
+    result = yield c.submit(d.get, 'a')
+    assert result == 1
+
+
 if sys.version_info >= (3, 5):
     from distributed.tests.py3_test_client import *  # flake8: noqa
Repository: dask/distributed

Issue:
lz4 new api
I believe the code here is faulty: https://github.com/dask/distributed/blob/cc908e7725410278c3a2ba08bd05675c186479e1/distributed/protocol/compression.py#L56-66. You need to import lz4.block in order to use the new api. Right now this always fall back. This is how I would fix it.

```python
with ignoring(ImportError):
    import lz4

    try:
        # try using the new lz4 API
        import lz4.block
        lz4_compress = lz4.block.compress
        lz4_decompress = lz4.block.decompress
    except ImportError:
        # fall back to old one
        lz4_compress = lz4.LZ4_compress
        lz4_decompress = lz4.LZ4_uncompress


Discussion:
Indeed

```
In [1]: import lz4

In [2]: lz4.block.compress
Out[2]: <function lz4.block._block.compress>

In [3]: lz4.__version__
Out[3]: '0.10.0'
```

```python
In [1]: import lz4

In [2]: lz4.__version__
Out[2]: '0.23.1'

In [3]: lz4.block
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-3-125c2d072986> in <module>()
----> 1 lz4.block

AttributeError: module 'lz4' has no attribute 'block'
```

Would you be willing to submit a PR with a fix?

Resolving patch:
diff --git a/distributed/protocol/compression.py b/distributed/protocol/compression.py
--- a/distributed/protocol/compression.py
+++ b/distributed/protocol/compression.py
@@ -58,9 +58,10 @@ def _fixed_snappy_decompress(data):
 
     try:
         # try using the new lz4 API
+        import lz4.block
         lz4_compress = lz4.block.compress
         lz4_decompress = lz4.block.decompress
-    except AttributeError as err:
+    except ImportError:
         # fall back to old one
         lz4_compress = lz4.LZ4_compress
         lz4_decompress = lz4.LZ4_uncompress


Test patch: