internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: dask/distributed

Issue:
test_local_tls fails on master (MacOS High Sierra, Ubuntu)
Reproducer (on a fresh `Python3.7` virtual environment, assuming you at the `distributed` root)

```
pip install -e .
python -m pytest distributed/deploy -k test_local_tls
```

By bisecting the repository it seems that the faulty commit is 3151f09e.
I am not an expert, but it seems that there relying only on `hostname` is not enough -- itself probably being linked tot the fact that we don't pass the IP version (v4 or v6) anymore. As proposed by the already existent [inline comment](https://github.com/dask/distributed/blob/4081a0e09febd9099f04a1db9772f6cd95203b2e/distributed/utils.py#L141), we should probably use `socket.getaddrinfo` or `socket.getfqdn()` on top of the `hostname`.

This currently affects the downstream builds of `cloudpickle`, (see cloudpipe/cloudpickle#324), so pinging @jakirkham :)


Discussion:

Resolving patch:
diff --git a/distributed/utils.py b/distributed/utils.py
--- a/distributed/utils.py
+++ b/distributed/utils.py
@@ -138,13 +138,15 @@ def _get_ip(host, port, family):
         ip = sock.getsockname()[0]
         return ip
     except EnvironmentError as e:
-        # XXX Should first try getaddrinfo() on socket.gethostname() and getfqdn()
         warnings.warn(
             "Couldn't detect a suitable IP address for "
             "reaching %r, defaulting to hostname: %s" % (host, e),
             RuntimeWarning,
         )
-        return socket.gethostname()
+        addr_info = socket.getaddrinfo(
+            socket.gethostname(), port, family, socket.SOCK_DGRAM, socket.IPPROTO_UDP
+        )[0]
+        return addr_info[4][0]
     finally:
         sock.close()
 


Test patch:
Repository: dask/distributed

Issue:
change default value of deploy.cli.dask-worker to use .config/dask if not given
In case of using `dask-worker` from command line, the default value of local-directory is an empty string. If you want to define a default directory in `.config/dask.dask.yaml` by setting the `temporary-directory` value.
In `distributed/worker.py Worker` the configured value from config file will only be used if `local-directory is None` which is not an empty string.

The same is with some other options which should be `None` instead of `""` for consistency.


Discussion:

Resolving patch:
diff --git a/distributed/cli/dask_worker.py b/distributed/cli/dask_worker.py
--- a/distributed/cli/dask_worker.py
+++ b/distributed/cli/dask_worker.py
@@ -151,12 +151,12 @@
 )
 @click.option("--pid-file", type=str, default="", help="File to write the process PID")
 @click.option(
-    "--local-directory", default="", type=str, help="Directory to place worker files"
+    "--local-directory", default=None, type=str, help="Directory to place worker files"
 )
 @click.option(
     "--resources",
     type=str,
-    default="",
+    default=None,
     help='Resources for task constraints like "GPU=2 MEM=10e9". '
     "Resources are applied separately to each worker process "
     "(only relevant when starting multiple worker processes with '--nprocs').",
@@ -164,7 +164,7 @@
 @click.option(
     "--scheduler-file",
     type=str,
-    default="",
+    default=None,
     help="Filename to JSON encoded scheduler information. "
     "Use with dask-scheduler --scheduler-file",
 )
@@ -180,7 +180,7 @@
 @click.option(
     "--lifetime",
     type=str,
-    default="",
+    default=None,
     help="If provided, shut down the worker after this duration.",
 )
 @click.option(


Test patch:
Repository: dask/distributed

Issue:
Attempts to connet to a non-running server spam it with connect attempts
This error does not make sense, on the face of it – either you time out, implying that there was no answer, or the connection was rejected, implying that the server told you there's no listener.
```
OSError: Timed out trying to connect to 'tcp://127.0.0.1:60590' after 10 s: in <distributed.comm.tcp.TCPConnector object at 0x7fce383a1ef0>: ConnectionRefusedError: [Errno 111] Connection refused
```
What actually happens is that `distributed` spams the server with connect requests for ten seconds, all of which get rejected. This is not a good idea. At all. Please either use a reasonable back-off or simply believe the server when it says ECONNREFUSED.


Discussion:
Thank you for the issue @smurfix 

> Please either use a reasonable back-off 

This seems reasonable to me.  Is this something that you would be interested in contributing?

> or simply believe the server when it says ECONNREFUSED.

Sometimes the server comes online during the connection process.  This happens frequently in Kubernetes situations where the address genuinely may not have existed a short while ago.

Resolving patch:
diff --git a/distributed/comm/core.py b/distributed/comm/core.py
--- a/distributed/comm/core.py
+++ b/distributed/comm/core.py
@@ -202,6 +202,10 @@ def _raise(error):
         )
         raise IOError(msg)
 
+    backoff = 0.01
+    if timeout and timeout / 20 < backoff:
+        backoff = timeout / 20
+
     # This starts a thread
     while True:
         try:
@@ -221,8 +225,10 @@ def _raise(error):
         except EnvironmentError as e:
             error = str(e)
             if time() < deadline:
-                await asyncio.sleep(0.01)
-                logger.debug("sleeping on connect")
+                logger.debug("Could not connect, waiting before retrying")
+                await asyncio.sleep(backoff)
+                backoff *= 1.5
+                backoff = min(backoff, 1)  # wait at most one second
             else:
                 _raise(error)
         else:


Test patch:
Repository: dask/distributed

Issue:
Adding zoom tools to performance_report
In some cases ["the devil is in the detail"]( https://en.wikipedia.org/wiki/The_devil_is_in_the_detail ), it would be useful to have the zoom tooltips included in other [`performance_report`]( https://distributed.dask.org/en/latest/diagnosing-performance.html#performance-reports ) panels to allow closer investigation of where time is being spent.


Discussion:
Sounds great to me

On Mon, Mar 23, 2020 at 3:46 PM jakirkham <notifications@github.com> wrote:

> In some cases "the devil is in the detail"
> <https://en.wikipedia.org/wiki/The_devil_is_in_the_detail>, it would be
> useful to have the zoom tooltips included in other performance_report
> <https://distributed.dask.org/en/latest/diagnosing-performance.html#performance-reports>
> panels to allow closer investigation of where time is being spent.
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/3626>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTD2JILO3E2NBFUSFO3RI7RE7ANCNFSM4LSHEBAA>
> .
>


Resolving patch:
diff --git a/distributed/profile.py b/distributed/profile.py
--- a/distributed/profile.py
+++ b/distributed/profile.py
@@ -383,7 +383,7 @@ def plot_figure(data, **kwargs):
 
     source = ColumnDataSource(data=data)
 
-    fig = figure(tools="tap", **kwargs)
+    fig = figure(tools="tap,box_zoom,xwheel_zoom,reset", **kwargs)
     r = fig.quad(
         "left",
         "right",


Test patch:
Repository: dask/distributed

Issue:
remote_path is ignored in SSHCluster
Current State:

In the doc: https://docs.dask.org/en/latest/setup/ssh.html#cmdoption-dask-ssh-remote-python
It is not specified that one can use `remote_python` in SSHCluster, although both `worker_options` and `scheduler_options` support it:
- https://github.com/dask/distributed/blob/master/distributed/deploy/ssh.py#L87
- https://github.com/dask/distributed/blob/master/distributed/deploy/ssh.py#L163

Furthermore, these parameters are ignored when passed, dask always using `sys.executable` in both remote scheduler and workers.

Expected behavior:
I would like to pass `remote_python` to both `worker_options` and `scheduler_options`.

Implementation details:

- https://github.com/dask/distributed/blob/master/distributed/deploy/ssh.py#L113
- https://github.com/dask/distributed/blob/master/distributed/deploy/ssh.py#L189

Both calls should replaced with `self.remote_python or sys.executable`, as it is in [old_ssh.py](https://github.com/dask/distributed/blob/master/distributed/deploy/old_ssh.py#L295)



Discussion:

Resolving patch:
diff --git a/distributed/deploy/ssh.py b/distributed/deploy/ssh.py
--- a/distributed/deploy/ssh.py
+++ b/distributed/deploy/ssh.py
@@ -110,7 +110,7 @@ async def start(self):
         cmd = " ".join(
             [
                 set_env,
-                sys.executable,
+                self.remote_python or sys.executable,
                 "-m",
                 self.worker_module,
                 self.scheduler,
@@ -186,7 +186,12 @@ async def start(self):
                 )
 
         cmd = " ".join(
-            [set_env, sys.executable, "-m", "distributed.cli.dask_scheduler",]
+            [
+                set_env,
+                self.remote_python or sys.executable,
+                "-m",
+                "distributed.cli.dask_scheduler",
+            ]
             + cli_keywords(self.kwargs, cls=_Scheduler)
         )
         self.proc = await self.connection.create_process(cmd)


Test patch:
Repository: dask/distributed

Issue:
Typo in docs
Just noticed there is a typo here. It should say `maximum_memory : str`.

https://github.com/dask/distributed/blob/9622b8f9bef1855412e9b23265378e2da1f47f2f/distributed/deploy/spec.py#L554


Discussion:

Resolving patch:
diff --git a/distributed/deploy/spec.py b/distributed/deploy/spec.py
--- a/distributed/deploy/spec.py
+++ b/distributed/deploy/spec.py
@@ -551,7 +551,7 @@ def adapt(
         minimum_memory : str
             Minimum amount of memory to keep around in the cluster
             Expressed as a string like "100 GiB"
-        maximum_cores : int
+        maximum_memory : str
             Maximum amount of memory to keep around in the cluster
             Expressed as a string like "100 GiB"
 


Test patch:
Repository: dask/distributed

Issue:
Proposal: Update colour scheme for `bytes stored` dashboard component
Currently, the dashboard component that shows the number of bytes stored by each worker, has a simple colour scheme:
- red if it has exceeded the set limits
- orange if it has exceeded 50% of limits
- blue otherwise
https://github.com/dask/distributed/blob/95228dc2d480aa35dc6521cb46bdb34983583faa/distributed/dashboard/components/scheduler.py#L790-L795

I would find it more useful if the colours reflected where the workers sat compared to the `target`/`spill`/`pause`/`terminate` limits that can be set in Dask's configuration.

https://distributed.dask.org/en/latest/worker.html#memory-management

I would be happy to contribute a PR to change this behaviour if you agree this would be a useful change. (I would also appreciate it if you offered opinions on what colours to use)


Discussion:

Resolving patch:
diff --git a/distributed/dashboard/components/scheduler.py b/distributed/dashboard/components/scheduler.py
--- a/distributed/dashboard/components/scheduler.py
+++ b/distributed/dashboard/components/scheduler.py
@@ -664,14 +664,42 @@ def update(self):
                     getattr(self.scheduler.workers[ws.address], "memory_limit", inf)
                     or inf
                 )
+                pause = (
+                    getattr(
+                        self.scheduler.workers[ws.address],
+                        "memory_target_fraction",
+                        inf,
+                    )
+                    or inf
+                )
+                spill = (
+                    getattr(
+                        self.scheduler.workers[ws.address],
+                        "memory_target_fraction",
+                        inf,
+                    )
+                    or inf
+                )
+                target = (
+                    getattr(
+                        self.scheduler.workers[ws.address],
+                        "memory_target_fraction",
+                        inf,
+                    )
+                    or inf
+                )
 
                 if limit > max_limit and limit != inf:
                     max_limit = limit
 
                 if nb > limit:
+                    nbytes_color.append("black")
+                elif nb > pause:
                     nbytes_color.append("red")
-                elif nb > limit / 2:
+                elif nb > target:
                     nbytes_color.append("orange")
+                elif nb > target * 0.85:
+                    nbytes_color.append("yellow")
                 else:
                     nbytes_color.append("blue")
 


Test patch:
Repository: dask/distributed

Issue:
Add logo to performance_report browser tab
Dashboard pages have the Dask logo as a favicon for browser tabs

<img width="353" alt="Screen Shot 2019-09-26 at 9 01 32 AM" src="https://user-images.githubusercontent.com/11656932/65728164-b98ced80-e07f-11e9-9220-138a11c73555.png">

It would be nice if `performance_report` pages also had the logo as a favicon 


Discussion:
+1

On Wed, Apr 29, 2020 at 8:59 AM James Bourbeau <notifications@github.com>
wrote:

> Dashboard pages have the Dask logo as a favicon for browser tabs
>
> [image: Screen Shot 2019-09-26 at 9 01 32 AM]
> <https://user-images.githubusercontent.com/11656932/65728164-b98ced80-e07f-11e9-9220-138a11c73555.png>
>
> It would be nice if performance_report pages also had the logo as a
> favicon
>
> —
> You are receiving this because you are subscribed to this thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/distributed/issues/3754>, or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AACKZTA4WJSGTYOGYR32FPTRPBFGZANCNFSM4MT3FTLQ>
> .
>


Resolving patch:
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -5154,10 +5154,17 @@ def profile_to_figure(state):
         )
 
         from bokeh.plotting import save, output_file
+        from bokeh.core.templates import get_env
 
         with tmpfile(extension=".html") as fn:
             output_file(filename=fn, title="Dask Performance Report")
-            save(tabs, filename=fn)
+            template_directory = os.path.join(
+                os.path.dirname(os.path.abspath(__file__)), "dashboard", "templates"
+            )
+            template_environment = get_env()
+            template_environment.loader.searchpath.append(template_directory)
+            template = template_environment.get_template("performance_report.html")
+            save(tabs, filename=fn, template=template)
 
             with open(fn) as f:
                 data = f.read()


Test patch:
Repository: dask/distributed

Issue:
Default value for Security(require_encryption=) keyword
I'm curious about the `require_encryption=` keyword.  Currently it defaults to False.

I'm curious if it should default to True if the user provides TLS information

cc @jcrist 


Discussion:
Yeah, IMO this should always be true, I don't see a use case for enabling TLS but not requiring it. If a user wants to enable an encrypted channel and an unencrypted channel, they should listen on multiple ports.

Resolving patch:
diff --git a/distributed/security.py b/distributed/security.py
--- a/distributed/security.py
+++ b/distributed/security.py
@@ -60,13 +60,15 @@ class Security:
         "tls_worker_cert",
     )
 
-    def __init__(self, **kwargs):
+    def __init__(self, require_encryption=None, **kwargs):
         extra = set(kwargs).difference(self.__slots__)
         if extra:
             raise TypeError("Unknown parameters: %r" % sorted(extra))
-        self._set_field(
-            kwargs, "require_encryption", "distributed.comm.require-encryption"
-        )
+        if require_encryption is None:
+            require_encryption = dask.config.get("distributed.comm.require-encryption")
+        if require_encryption is None:
+            require_encryption = not not kwargs
+        self.require_encryption = require_encryption
         self._set_field(kwargs, "tls_ciphers", "distributed.comm.tls.ciphers")
         self._set_field(kwargs, "tls_ca_file", "distributed.comm.tls.ca-file")
         self._set_field(kwargs, "tls_client_key", "distributed.comm.tls.client.key")


Test patch:
diff --git a/distributed/tests/test_security.py b/distributed/tests/test_security.py
--- a/distributed/tests/test_security.py
+++ b/distributed/tests/test_security.py
@@ -109,7 +109,7 @@ def test_repr():
     sec = Security(tls_ca_file="ca.pem", tls_scheduler_cert="scert.pem")
     assert (
         repr(sec)
-        == "Security(require_encryption=False, tls_ca_file='ca.pem', tls_scheduler_cert='scert.pem')"
+        == "Security(require_encryption=True, tls_ca_file='ca.pem', tls_scheduler_cert='scert.pem')"
     )
 
 
diff --git a/distributed/tests/test_tls_functional.py b/distributed/tests/test_tls_functional.py
--- a/distributed/tests/test_tls_functional.py
+++ b/distributed/tests/test_tls_functional.py
@@ -201,11 +201,14 @@ async def test_security_dict_input(cleanup):
     scheduler = conf["distributed"]["comm"]["tls"]["scheduler"]["cert"]
 
     async with Scheduler(
-        security={"tls_ca_file": ca_file, "tls_scheduler_cert": scheduler}
+        host="localhost",
+        security={"tls_ca_file": ca_file, "tls_scheduler_cert": scheduler},
     ) as s:
+        assert s.address.startswith("tls://")
         async with Worker(
             s.address, security={"tls_ca_file": ca_file, "tls_worker_cert": worker}
         ) as w:
+            assert w.address.startswith("tls://")
             async with Client(
                 s.address,
                 security={"tls_ca_file": ca_file, "tls_client_cert": client},
Repository: dask/distributed

Issue:
Variables leak virtual clients
**What happened**:

```python
def _get_number_of_clients(dask_scheduler: Optional[Scheduler] = None) -> Optional[int]:
    if dask_scheduler is None:
        return None
    else:
        return len(dask_scheduler.clients)

n_clients1 = client.run_on_scheduler(_get_number_of_clients)
assert n_clients1 is not None

for i in range(100):
    future = create_some_dummy_future()
    var = Variable(f"var-{i}", client)
    var.set(future)
    future.cancel(force=True)
    var.delete()

n_clients2 = client.run_on_scheduler(_get_number_of_clients)
assert n_clients2 is not None

assert n_client1 == n_clients2
```

This fails, because here:

https://github.com/dask/distributed/blob/1d7640b0172febf9ceef37c2c31241c66ac165eb/distributed/scheduler.py#L2333-L2347

a new virtual client for each and every variable is created but here:

https://github.com/dask/distributed/blob/1d7640b0172febf9ceef37c2c31241c66ac165eb/distributed/scheduler.py#L2349-L2361

we never clean up / prune these clients again.

**What you expected to happen**:

Scheduler-side state is not leaky.

**Environment**:

- Dask version: 2.17.2
- Distributed version: 2.17.0
- Python version: 3.6
- Operating System: Linux
- Install method (conda, pip, source): pip



Discussion:
Thank you for the issue @marco-neumann-jdas .  I'm able to reproduce.  

It looks like we should call `Scheduler.remove_client` when we delete a variable (assuming that it has already been created).  Is this something that you would be interested in fixing?

Resolving patch:
diff --git a/distributed/variable.py b/distributed/variable.py
--- a/distributed/variable.py
+++ b/distributed/variable.py
@@ -119,6 +119,8 @@ async def delete(self, comm=None, name=None, client=None):
             with suppress(KeyError):
                 del self.variables[name]
 
+            self.scheduler.remove_client("variable-%s" % name)
+
 
 class Variable:
     """ Distributed Global Variable


Test patch:
diff --git a/distributed/tests/test_variable.py b/distributed/tests/test_variable.py
--- a/distributed/tests/test_variable.py
+++ b/distributed/tests/test_variable.py
@@ -263,3 +263,22 @@ def test_future_erred_sync(client):
 
     with pytest.raises(ZeroDivisionError):
         future2.result()
+
+
+@gen_cluster(client=True)
+async def test_variables_do_not_leak_client(c, s, a, b):
+    # https://github.com/dask/distributed/issues/3899
+    clients_pre = set(s.clients)
+
+    # setup variable with future
+    x = Variable("x")
+    future = c.submit(inc, 1)
+    await x.set(future)
+
+    # complete teardown
+    x.delete()
+
+    start = time()
+    while set(s.clients) != clients_pre:
+        await asyncio.sleep(0.01)
+        assert time() < start + 5
Repository: dask/distributed

Issue:
Dashboard issues with very large clusters
I'm on the latest conda defaults channel  `dask` and `distributed` and see formatting issues of the dashboard with very large clusters:

- On `...:8787/workers`: For a cluster of 100 workers with 24 maxed out (each at 100% cpu) threads each I get total CPU displayed as roughly the average use of one worker (2400%)

![image](https://user-images.githubusercontent.com/5700886/89029134-08558480-d32e-11ea-8bd8-235efb004a28.png)

- On `...:8787/status`: The histogram view of memory use displays bytes stored of O(1 MB) while the true memory occupation of the cluster is O(10TB):

![image](https://user-images.githubusercontent.com/5700886/89029328-77cb7400-d32e-11ea-9a64-8c759079605e.png)


Discussion:
My guess is that we're using the `dask.sizeof.sizeof` function to guess the size in bytes of all of the data tracked by each worker, and that we're underestimating the objects that you're using.  

For the non-histogram plot (which is used for fewer workers) we switched from using sizeof to using the memory of the process as reported by psutil.  We should probably make this same change for the histogram plot.

I think that we probably want to replace the `ws.nbytes` here

https://github.com/dask/distributed/blob/7cdba1d7e17df7be005f4d4b8ca528065a3f83a7/distributed/dashboard/components/scheduler.py#L264-L271

Any interest in handling this @willirath ?  It seems like a mostly easy fix.
> Any interest in handling this @willirath ? It seems like a mostly easy fix.

Yes, I'll have a look next week.




Resolving patch:
diff --git a/distributed/dashboard/components/scheduler.py b/distributed/dashboard/components/scheduler.py
--- a/distributed/dashboard/components/scheduler.py
+++ b/distributed/dashboard/components/scheduler.py
@@ -263,7 +263,9 @@ def __init__(self, scheduler, **kwargs):
 
     @without_property_validation
     def update(self):
-        nbytes = np.asarray([ws.nbytes for ws in self.scheduler.workers.values()])
+        nbytes = np.asarray(
+            [ws.metrics["memory"] for ws in self.scheduler.workers.values()]
+        )
         counts, x = np.histogram(nbytes, bins=40)
         d = {"left": x[:-1], "right": x[1:], "top": counts}
         self.source.data.update(d)


Test patch:
diff --git a/distributed/dashboard/tests/test_scheduler_bokeh.py b/distributed/dashboard/tests/test_scheduler_bokeh.py
--- a/distributed/dashboard/tests/test_scheduler_bokeh.py
+++ b/distributed/dashboard/tests/test_scheduler_bokeh.py
@@ -276,7 +276,7 @@ async def test_ProcessingHistogram(c, s, a, b):
 async def test_NBytesHistogram(c, s, a, b):
     nh = NBytesHistogram(s)
     nh.update()
-    assert (nh.source.data["top"] != 0).sum() == 1
+    assert any(nh.source.data["top"] != 0)
 
     futures = c.map(inc, range(10))
     await wait(futures)