Repository: dask/distributed

Issue:
Version mismatch warning is a little scary
## Background

When clients/scheduler/workers have mismatched versions, users get an informative error message like the following:

```
/home/mrocklin/workspace/distributed/distributed/client.py:1079: VersionMismatchWarning: Mismatched versions found

blosc
+---------------------------+---------+
|                           | version |
+---------------------------+---------+
| client                    | None    |
| scheduler                 | 1.9.1   |
| tcp://172.31.15.170:46853 | 1.9.1   |
| tcp://172.31.18.92:41153  | 1.9.1   |
| tcp://172.31.42.33:42009  | 1.9.1   |
| tcp://172.31.7.159:38461  | 1.9.1   |
+---------------------------+---------+

cloudpickle
+---------------------------+---------+
|                           | version |
+---------------------------+---------+
| client                    | 1.3.0   |
| scheduler                 | 1.4.0   |
| tcp://172.31.15.170:46853 | 1.4.0   |
| tcp://172.31.18.92:41153  | 1.4.0   |
| tcp://172.31.42.33:42009  | 1.4.0   |
| tcp://172.31.7.159:38461  | 1.4.0   |
+---------------------------+---------+

dask
+---------------------------+---------------------+
|                           | version             |
+---------------------------+---------------------+
| client                    | 2.14.0+34.g8ab7f942 |
| scheduler                 | 2.15.0              |
| tcp://172.31.15.170:46853 | 2.15.0              |
| tcp://172.31.18.92:41153  | 2.15.0              |
| tcp://172.31.42.33:42009  | 2.15.0              |
| tcp://172.31.7.159:38461  | 2.15.0              |
+---------------------------+---------------------+

distributed
+---------------------------+---------------------+
|                           | version             |
+---------------------------+---------------------+
| client                    | 2.14.0+47.gb4dc9c64 |
| scheduler                 | 2.15.0              |
| tcp://172.31.15.170:46853 | 2.15.0              |
| tcp://172.31.18.92:41153  | 2.15.0              |
| tcp://172.31.42.33:42009  | 2.15.0              |
| tcp://172.31.7.159:38461  | 2.15.0              |
+---------------------------+---------------------+

lz4
+---------------------------+---------+
|                           | version |
+---------------------------+---------+
| client                    | 2.2.1   |
| scheduler                 | 3.0.2   |
| tcp://172.31.15.170:46853 | 3.0.2   |
| tcp://172.31.18.92:41153  | 3.0.2   |
| tcp://172.31.42.33:42009  | 3.0.2   |
| tcp://172.31.7.159:38461  | 3.0.2   |
+---------------------------+---------+

msgpack
+---------------------------+---------+
|                           | version |
+---------------------------+---------+
| client                    | 0.6.2   |
| scheduler                 | 1.0.0   |
| tcp://172.31.15.170:46853 | 1.0.0   |
| tcp://172.31.18.92:41153  | 1.0.0   |
| tcp://172.31.42.33:42009  | 1.0.0   |
| tcp://172.31.7.159:38461  | 1.0.0   |
+---------------------------+---------+

python
+---------------------------+---------------+
|                           | version       |
+---------------------------+---------------+
| client                    | 3.7.6.final.0 |
| scheduler                 | 3.7.4.final.0 |
| tcp://172.31.15.170:46853 | 3.7.4.final.0 |
| tcp://172.31.18.92:41153  | 3.7.4.final.0 |
| tcp://172.31.42.33:42009  | 3.7.4.final.0 |
| tcp://172.31.7.159:38461  | 3.7.4.final.0 |
+---------------------------+---------------+
  warnings.warn(version_module.VersionMismatchWarning(msg[0]["warning"]))
```

This is generally pretty great.  We used to get a ton of github issues that reduced down to version mismatches, and now we don't.  Hooray for informative error messages.

## Moving forward

However, I've run into a couple of issues that arise with these messages in practice, where I think that we might be able to improve them a bit.

1.  They can get very long, especially if you have lots of workers.
2.  We don't call out really important issues in relation to less important issues.  It's entirely ok if your `msgpack` version is a little off, but probably not ok if some machines have `lz4` and some don't.  

So I wonder if we might reorganize this message a bit.  We might have something like the following:

```
+-----------+----------+------------+------------+
| Package   |  client  |  scheduler |    workers |
+-----------+----------+------------+------------+
| python    |    3.7   |    3.8     | {3.7, 3.8} |
| lz4       |   ...    |    ...     |     ...    |
| msgpack   |          |    ...     |     ...    |
+-----------+----------+------------+------------+

Notes:

-  msgpack: Variation is ok, as long as everything is above 0.6
-  lz4: Variation is ok, but missing libraries are not
-  python: Variation is sometimes ok, sometimes not.  It depends on your workloads
```

We pack down the one-line-per-worker policy, and instead include sets of versions in the table if necessary.  This makes it a bit harder to debug if one of the workers is mismatched, but I think that this behavior isn't common.  

We include optional prose around each of the libraries and include that prose if that library is found to be mismatched.


Discussion:
Thanks for raising this @mrocklin. I'm generally in favor of moving to a one-line-per-worker policy and a big +1 from me for adding optional prose for libraries to provide more context to users. If it's not too much more work, we could also add a `verbose=` option (`False` by default) to `Client.get_versions` where the current output is produced for cases where that's useful for debugging
This change definitely sounds better than the status-quo. I think I'm a little more in favor of not warning at all for things that are probably fine (e.g. msgpack version as mentioned above). Environment mismatches are not uncommon, but usually are fine - having the standard dask user session polluted by a big warning on startup can be off-putting even if you say "don't worry, this is fine" (it doesn't help that notebooks display warnings in red, which implicitly makes things look "scary").

Resolving patch:
diff --git a/distributed/versions.py b/distributed/versions.py
--- a/distributed/versions.py
+++ b/distributed/versions.py
@@ -2,7 +2,6 @@
 
 from __future__ import print_function, division, absolute_import
 
-from collections import defaultdict
 import platform
 import struct
 import os
@@ -32,6 +31,14 @@
 )
 
 
+# notes to be displayed for mismatch packages
+notes_mismatch_package = {
+    "msgpack": "Variation is ok, as long as everything is above 0.6",
+    "lz4": "Variation is ok, but missing libraries are not",
+    "python": "Variation is sometimes ok, sometimes not. It depends on your workloads",
+}
+
+
 def get_versions(packages=None):
     """
     Return basic information on our software installation, and our installed versions of packages.
@@ -106,37 +113,56 @@ def get_package_info(pkgs):
 def error_message(scheduler, workers, client, client_name="client"):
     from .utils import asciitable
 
-    nodes = {**{client_name: client}, **{"scheduler": scheduler}, **workers}
-
-    # Hold all versions, e.g. versions["scheduler"]["distributed"] = 2.9.3
-    node_packages = defaultdict(dict)
+    client = client.get("packages") if client else "UNKNOWN"
+    scheduler = scheduler.get("packages") if scheduler else "UNKNOWN"
+    workers = {k: v.get("packages") if v else "UNKNOWN" for k, v in workers.items()}
 
-    # Collect all package versions
     packages = set()
-    for node, info in nodes.items():
-        if info is None or not (isinstance(info, dict)) or "packages" not in info:
-            node_packages[node] = defaultdict(lambda: "UNKNOWN")
-        else:
-            node_packages[node] = defaultdict(lambda: "MISSING")
-            for pkg, version in info["packages"].items():
-                node_packages[node][pkg] = version
-                packages.add(pkg)
+    packages.update(client)
+    packages.update(scheduler)
+    for worker in workers:
+        packages.update(workers.get(worker))
 
     errs = []
+    notes = []
     for pkg in sorted(packages):
-        versions = set(
-            node_packages[node][pkg]
-            for node in nodes
-            if node != "scheduler" or pkg in scheduler_relevant_packages
+        versions = set()
+        scheduler_version = (
+            scheduler.get(pkg, "MISSING") if isinstance(scheduler, dict) else scheduler
+        )
+        if pkg in scheduler_relevant_packages:
+            versions.add(scheduler_version)
+
+        client_version = (
+            client.get(pkg, "MISSING") if isinstance(client, dict) else client
         )
+        versions.add(client_version)
+
+        worker_versions = set(
+            workers[w].get(pkg, "MISSING")
+            if isinstance(workers[w], dict)
+            else workers[w]
+            for w in workers
+        )
+        versions |= worker_versions
+
         if len(versions) <= 1:
             continue
-        rows = [
-            (node_name, node_packages[node_name][pkg]) for node_name in nodes.keys()
-        ]
-        errs.append("%s\n%s" % (pkg, asciitable(["", "version"], rows)))
+        if len(worker_versions) == 1:
+            worker_versions = list(worker_versions)[0]
+        elif len(worker_versions) == 0:
+            worker_versions = None
+
+        errs.append((pkg, client_version, scheduler_version, worker_versions))
+        if pkg in notes_mismatch_package.keys():
+            notes.append(f"-  {pkg}: {notes_mismatch_package[pkg]}")
+
     if errs:
-        return "Mismatched versions found\n" "\n" "%s" % ("\n\n".join(errs))
+        err_table = asciitable(["Package", client_name, "scheduler", "workers"], errs)
+        err_msg = f"Mismatched versions found\n\n{err_table}"
+        if notes:
+            err_msg += "\nNotes: \n{}".format("\n".join(notes))
+        return err_msg
     else:
         return ""
 


Test patch:
diff --git a/distributed/tests/test_versions.py b/distributed/tests/test_versions.py
--- a/distributed/tests/test_versions.py
+++ b/distributed/tests/test_versions.py
@@ -67,9 +67,9 @@ def kwargs_not_matching(kwargs_matching, node, effect):
 
 @pytest.fixture
 def pattern(effect):
-    """Pattern to match in the right-hand column."""
+    """String to match in the right column."""
     return {
-        "MISMATCHED": r"0\.0\.0\.dev0",
+        "MISMATCHED": "0.0.0.dev0",
         "MISSING": "MISSING",
         "KEY_ERROR": "UNKNOWN",
         "NONE": "UNKNOWN",
@@ -77,11 +77,18 @@ def pattern(effect):
 
 
 def test_version_mismatch(node, effect, kwargs_not_matching, pattern):
+    column_matching = {"client": 1, "scheduler": 2, "workers": 3}
     msg = error_message(**kwargs_not_matching)
-
+    i = column_matching.get(node, 3)
     assert "Mismatched versions found" in msg
     assert "distributed" in msg
-    assert re.search(node + r"\s+\|\s+" + pattern, msg)
+    assert (
+        pattern
+        in re.search(r"distributed\s+(?:(?:\|[^|\r\n]*)+\|(?:\r?\n|\r)?)+", msg)
+        .group(0)
+        .split("|")[i]
+        .strip()
+    )
 
 
 def test_scheduler_mismatched_irrelevant_package(kwargs_matching):
@@ -99,6 +106,20 @@ def test_scheduler_additional_irrelevant_package(kwargs_matching):
     assert error_message(**kwargs_matching) == ""
 
 
+def test_python_mismatch(kwargs_matching):
+    kwargs_matching["client"]["packages"]["python"] = "0.0.0"
+    msg = error_message(**kwargs_matching)
+    assert "Mismatched versions found" in msg
+    assert "python" in msg
+    assert (
+        "0.0.0"
+        in re.search(r"python\s+(?:(?:\|[^|\r\n]*)+\|(?:\r?\n|\r)?)+", msg)
+        .group(0)
+        .split("|")[1]
+        .strip()
+    )
+
+
 @gen_cluster()
 async def test_version_warning_in_cluster(s, a, b):
     s.workers[a.address].versions["packages"]["dask"] = "0.0.0"
@@ -110,14 +131,11 @@ async def test_version_warning_in_cluster(s, a, b):
     assert record
     assert any("dask" in str(r.message) for r in record)
     assert any("0.0.0" in str(r.message) for r in record)
-    assert any(a.address in str(r.message) for r in record)
 
     async with Worker(s.address) as w:
-        assert any("This Worker" in line.message for line in w.logs)
+        assert any("workers" in line.message for line in w.logs)
         assert any("dask" in line.message for line in w.logs)
-        assert any(
-            "0.0.0" in line.message and a.address in line.message for line in w.logs
-        )
+        assert any("0.0.0" in line.message in line.message for line in w.logs)
 
 
 def test_python_version():