Repository: dask/distributed

Issue:
Style hover tooltip in worker memory plot
The new managed worker memory is great.  I wonder if we could improve it by styling the hover tooltip a bit.  Currently the hover changes depending on if you're on the managed or unmanaged part.  It's also quite long, looking like this:

    tls://worker-address:1234: 10.0 GiB (8.33 GiB managed (in memory))

And then if you wander over to the right a bit you get something like 

    tls://worker-address:1234: 10.0 GiB (512 MiB unmanaged )

and then again for spilled to disk, etc..

These are long, and it's hard to see everything at a glance.  

Instead, we might consider using a styled table (without borders), maybe similar to how we do with the progress bar.  This includes all measurements at once, and uses bold/emphasized text to separate categories from values.  You don't need to trace your mouse over the horizontal bar to see the different values, which can be especially tricky during computation when the values move.  

We probably also want to avoid the full explanations in the tooltip, and instead find some name for these and put the description in documentation.




Discussion:
For convenience, here is the code for the hover for the progress bar.  I think that it might be a good model here.

https://github.com/dask/distributed/blob/26a1da050636b4e37490b675ba16aa29a484e106/distributed/dashboard/components/scheduler.py#L1825-L1849
> We probably also want to avoid the full explanations in the tooltip, and instead find some name for these and put the description in documentation.

xrefing https://github.com/dask/distributed/issues/4746
For reference the hover we want to modify is this one:
https://github.com/dask/distributed/blob/26a1da050636b4e37490b675ba16aa29a484e106/distributed/dashboard/components/scheduler.py#L379-L382
@mrocklin and @jrbourbeau In terms of style, how do we like this for the Hover? 
Showing screenshots of same computation, showing managed and unmanaged part.

![Screen Shot 2021-06-22 at 11 13 05 AM](https://user-images.githubusercontent.com/7526622/122951773-8908df00-d34b-11eb-877f-6bce41baeaac.png)
![Screen Shot 2021-06-22 at 11 13 26 AM](https://user-images.githubusercontent.com/7526622/122951786-8b6b3900-d34b-11eb-95c9-c90f452028f2.png)

There are several numbers to track (I think four or five?) I'll suggest a bit more structure.  Maybe something like the following:

## .

tcp://...

  **Total**: 100 GiB
  **Managed**: 70 GiB
  **Unmanaged**: 10 GiB
  **Pending**: 10 GiB
  **Disk**: 10 GIB

If there was some way to align the GB totals I think that that would be ideal.  Maybe a table without boundaries?
  
cc also @crusaderky 
an HTML multi-line tooltip was my first prototype for that PR. However I had to discard it because I could not find a way to fix an issue where the tooltip would end up hidden under other objects - namely the top bar.
One thing I notices is that what is displayed in the Hover the part that says for example `unmanaged: 305.66 MiB` is related to the `width` variable, which is what changes depending on where you hover. I believe the way it's designed we don't have in variables the breakdown to display something like. Maybe I'm wrong, @crusaderky ? 

Managed: 70 GiB
Unmanaged: 10 GiB
Pending: 10 GiB
Disk: 10 GIB

I'm hopeful that we can do whatever we did for the progress bar, which seems to accomplish this in a similar situation.
How about this? (I'm not sure what's the Pending status, but if we have that info in `meminfo` then I can display it too)

![Screen Shot 2021-06-22 at 12 36 42 PM](https://user-images.githubusercontent.com/7526622/122965949-c32bae00-d356-11eb-9f68-b8a1604c5d82.png)

Never heard of pending. I don't like the label "total", as it's ambiguous. The tooltip should show the process memory (which does not include the spilled memory).

We should strive for coherent naming between
- this tooltip
- the column headers in the workers tab
- the sphinx documentation
- ``MemoryState.__repr__``

Also, take a look at the latter for verbiage. Not saying that it should be copied verbatim, but I think that the tooltip and the repr should be kept aligned:
https://github.com/dask/distributed/blob/7d0f0102ddfec1195257c82889ff695d32d61fbc/distributed/scheduler.py#L370-L379


More importantly: if memory serves me, if you have many workers (e.g. 32) and hover on top of the first worker, the tooltip will be hidden by the top bar. The tooltip for the last workers will also be partially covered by whatever's below.

Resolving patch:
diff --git a/distributed/dashboard/components/scheduler.py b/distributed/dashboard/components/scheduler.py
--- a/distributed/dashboard/components/scheduler.py
+++ b/distributed/dashboard/components/scheduler.py
@@ -240,15 +240,13 @@ def __init__(self, scheduler, width=600, **kwargs):
                     "width": [0] * 4,
                     "x": [0] * 4,
                     "y": [0] * 4,
-                    "proc_memory": [0] * 4,
                     "color": ["blue", "blue", "blue", "grey"],
                     "alpha": [1, 0.7, 0.4, 1],
-                    "memtype": [
-                        "managed (in memory)",
-                        "unmanaged",
-                        "unmanaged, recently increased",
-                        "managed (spilled to disk)",
-                    ],
+                    "proc_memory": [0] * 4,
+                    "managed": [0] * 4,
+                    "unmanaged_old": [0] * 4,
+                    "unmanaged_recent": [0] * 4,
+                    "spilled": [0] * 4,
                 }
             )
 
@@ -283,9 +281,31 @@ def __init__(self, scheduler, width=600, **kwargs):
             self.root.toolbar_location = None
             self.root.yaxis.visible = False
 
-            hover = HoverTool()
-            hover.tooltips = "@width{0.00 b} @memtype"
-            hover.point_policy = "follow_mouse"
+            hover = HoverTool(
+                point_policy="follow_mouse",
+                tooltips="""
+                            <div>
+                                <span style="font-size: 12px; font-weight: bold;">Process memory (RSS):</span>&nbsp;
+                                <span style="font-size: 10px; font-family: Monaco, monospace;">@proc_memory{0.00 b}</span>
+                            </div>
+                            <div style="margin-left: 1em;">
+                                <span style="font-size: 12px; font-weight: bold;">Managed:</span>&nbsp;
+                                <span style="font-size: 10px; font-family: Monaco, monospace;">@managed{0.00 b}</span>
+                            </div>
+                            <div style="margin-left: 1em;">
+                                <span style="font-size: 12px; font-weight: bold;">Unmanaged (old):</span>&nbsp;
+                                <span style="font-size: 10px; font-family: Monaco, monospace;">@unmanaged_old{0.00 b}</span>
+                            </div>
+                            <div style="margin-left: 1em;">
+                                <span style="font-size: 12px; font-weight: bold;">Unmanaged (recent):</span>&nbsp;
+                                <span style="font-size: 10px; font-family: Monaco, monospace;">@unmanaged_recent{0.00 b}</span>
+                            </div>
+                            <div>
+                                <span style="font-size: 12px; font-weight: bold;">Spilled to disk:</span>&nbsp;
+                                <span style="font-size: 10px; font-family: Monaco, monospace;">@spilled{0.00 b}</span>
+                            </div>
+                            """,
+            )
             self.root.add_tools(hover)
 
     @without_property_validation
@@ -301,11 +321,16 @@ def update(self):
                 meminfo.unmanaged_recent,
                 meminfo.managed_spilled,
             ]
+
             result = {
                 "width": width,
                 "x": [sum(width[:i]) + w / 2 for i, w in enumerate(width)],
                 "color": [color, color, color, "grey"],
                 "proc_memory": [meminfo.process] * 4,
+                "managed": [meminfo.managed_in_memory] * 4,
+                "unmanaged_old": [meminfo.unmanaged_old] * 4,
+                "unmanaged_recent": [meminfo.unmanaged_recent] * 4,
+                "spilled": [meminfo.managed_spilled] * 4,
             }
             # FIXME https://github.com/dask/distributed/issues/4675
             #       This causes flickering after adding workers and when enough memory
@@ -333,12 +358,15 @@ def __init__(self, scheduler, width=600, **kwargs):
                     "width": [],
                     "x": [],
                     "y": [],
-                    "memtype": [],
                     "color": [],
                     "alpha": [],
-                    "proc_memory": [],
                     "worker": [],
                     "escaped_worker": [],
+                    "proc_memory": [],
+                    "managed": [],
+                    "unmanaged_old": [],
+                    "unmanaged_recent": [],
+                    "spilled": [],
                 }
             )
 
@@ -377,9 +405,35 @@ def __init__(self, scheduler, width=600, **kwargs):
             self.root.toolbar_location = None
             self.root.yaxis.visible = False
 
-            hover = HoverTool()
-            hover.tooltips = "@worker: @proc_memory{0.00 b} (@width{0.00 b} @memtype)"
-            hover.point_policy = "follow_mouse"
+            hover = HoverTool(
+                point_policy="follow_mouse",
+                tooltips="""
+                            <div>
+                                <span style="font-size: 12px; font-weight: bold;">Worker:</span>&nbsp;
+                                <span style="font-size: 10px; font-family: Monaco, monospace;">@worker</span>
+                            </div>
+                            <div>
+                                <span style="font-size: 12px; font-weight: bold;">Process memory (RSS):</span>&nbsp;
+                                <span style="font-size: 10px; font-family: Monaco, monospace;">@proc_memory{0.00 b}</span>
+                            </div>
+                            <div style="margin-left: 1em;">
+                                <span style="font-size: 12px; font-weight: bold;">Managed:</span>&nbsp;
+                                <span style="font-size: 10px; font-family: Monaco, monospace;">@managed{0.00 b}</span>
+                            </div>
+                            <div style="margin-left: 1em;">
+                                <span style="font-size: 12px; font-weight: bold;">Unmanaged (old):</span>&nbsp;
+                                <span style="font-size: 10px; font-family: Monaco, monospace;">@unmanaged_old{0.00 b}</span>
+                            </div>
+                            <div style="margin-left: 1em;">
+                                <span style="font-size: 12px; font-weight: bold;">Unmanaged (recent):</span>&nbsp;
+                                <span style="font-size: 10px; font-family: Monaco, monospace;">@unmanaged_recent{0.00 b}</span>
+                            </div>
+                            <div>
+                                <span style="font-size: 12px; font-weight: bold;">Spilled to disk:</span>&nbsp;
+                                <span style="font-size: 10px; font-family: Monaco, monospace;">@spilled{0.00 b}</span>
+                            </div>
+                            """,
+            )
             self.root.add_tools(hover)
 
     @without_property_validation
@@ -396,8 +450,12 @@ def quadlist(i) -> list:
             width = []
             x = []
             color = []
-            procmemory = []
             max_limit = 0
+            procmemory = []
+            managed = []
+            spilled = []
+            unmanaged_old = []
+            unmanaged_recent = []
 
             for ws in workers:
                 meminfo = ws.memory
@@ -415,26 +473,29 @@ def quadlist(i) -> list:
                 ]
                 x += [sum(width[-4:i]) + width[i] / 2 for i in range(-4, 0)]
                 color += [color_i, color_i, color_i, "grey"]
+
+                # memory info
                 procmemory.append(meminfo.process)
+                managed.append(meminfo.managed_in_memory)
+                unmanaged_old.append(meminfo.unmanaged_old)
+                unmanaged_recent.append(meminfo.unmanaged_recent)
+                spilled.append(meminfo.managed_spilled)
 
             result = {
                 "width": width,
                 "x": x,
                 "color": color,
                 "alpha": [1, 0.7, 0.4, 1] * len(workers),
-                "memtype": [
-                    "managed (in memory)",
-                    "unmanaged",
-                    "unmanaged, recently increased",
-                    "managed (spilled to disk)",
-                ]
-                * len(workers),
-                "proc_memory": quadlist(procmemory),
                 "worker": quadlist(ws.address for ws in workers),
                 "escaped_worker": quadlist(
                     escape.url_escape(ws.address) for ws in workers
                 ),
                 "y": quadlist(range(len(workers))),
+                "proc_memory": quadlist(procmemory),
+                "managed": quadlist(managed),
+                "unmanaged_old": quadlist(unmanaged_old),
+                "unmanaged_recent": quadlist(unmanaged_recent),
+                "spilled": quadlist(spilled),
             }
             # Remove rectangles with width=0
             result = {
@@ -1963,7 +2024,7 @@ def __init__(self, scheduler, width=800, **kwargs):
             "memory_limit": "limit",
             "memory_percent": "memory %",
             "memory_managed": "managed",
-            "memory_unmanaged_old": "unmanaged",
+            "memory_unmanaged_old": "unmanaged old",
             "memory_unmanaged_recent": "unmanaged recent",
             "memory_spilled": "spilled",
             "num_fds": "# fds",
diff --git a/distributed/scheduler.py b/distributed/scheduler.py
--- a/distributed/scheduler.py
+++ b/distributed/scheduler.py
@@ -369,13 +369,11 @@ def optimistic(self) -> Py_ssize_t:
 
     def __repr__(self) -> str:
         return (
-            f"Managed by Dask       : {format_bytes(self.managed)}\n"
-            f"  - in process memory : {format_bytes(self._managed_in_memory)}\n"
-            f"  - spilled to disk   : {format_bytes(self._managed_spilled)}\n"
             f"Process memory (RSS)  : {format_bytes(self._process)}\n"
             f"  - managed by Dask   : {format_bytes(self._managed_in_memory)}\n"
             f"  - unmanaged (old)   : {format_bytes(self._unmanaged_old)}\n"
             f"  - unmanaged (recent): {format_bytes(self.unmanaged_recent)}\n"
+            f"Spilled to disk       : {format_bytes(self._managed_spilled)}\n"
         )
 
 


Test patch:
diff --git a/distributed/tests/test_scheduler.py b/distributed/tests/test_scheduler.py
--- a/distributed/tests/test_scheduler.py
+++ b/distributed/tests/test_scheduler.py
@@ -2290,13 +2290,11 @@ def test_memorystate():
         repr(m)
         == dedent(
             """
-            Managed by Dask       : 80 B
-              - in process memory : 68 B
-              - spilled to disk   : 12 B
             Process memory (RSS)  : 100 B
               - managed by Dask   : 68 B
               - unmanaged (old)   : 15 B
               - unmanaged (recent): 17 B
+            Spilled to disk       : 12 B
             """
         ).lstrip()
     )