Repository: dask/dask

Issue:
Profile function naming
The profile currently prints names a bit oddly.  In particular

```
(f, (g,))  -> f

(lambda x: x,)  -> &lt;lambda&gt;
```

The first is also a problem in dot printing.



Discussion:
I wrote something up, but I'm not sure if it's a good solution. Best way is to show some examples.

``` python
import numpy as np
import dask.array as da

x = np.arange(100).reshape((10, 10))
a = da.from_array(x, chunks=(5, 5))
res = (a.mean(axis=0).sum() + 4)
res._visualize()
```

![mydask](https://cloud.githubusercontent.com/assets/2783717/8735514/fb49ffc2-2bd6-11e5-99ab-212e6539a730.png)

This shows the _entire_ task, with `_` where keys go, and types of constants if their repr is too long. This may work out for profiling, but for dot graphs, the labels are too long (IMO).

I agree with you that the length of the label disrupts the visual effect of the dot graph.  It's fairly easy to fix that though with the following within dot

``` python
s = name(task)
if len(s) > 10:
    s = s[:10] + '...'
```

For profiling though we can probably skip that curtailment.  

Thoughts on moving the paren to the right side of the function call?

What about adding line breaks to the labels in the dot graph?

I don't like that fix, that could result in truncated function names, missing close parens, etc... I'd prefer to just display the name of the top-level function for the task for the graph. Perhaps if there's nesting, something like `(foo, (...))` could work. For profiling though, I think this is good.

> Thoughts on moving the paren to the right side of the function call?

Could be done. I did it this way to match the internals of dask. No strong opinions here.

> What about adding line breaks to the labels in the dot graph?

I'm against doing this - the issue isn't that the circles are big, but that the text is too long to get a general sense of the flow. Also, some tasks (read_csv for example) create huge tasks, which will still be a problem even with line wrapping.

I see truncated function names and missing close parens as less bad than completely missing function names.  Often for nested tasks the important function is actually fairly deep within the task and the top-level function is somewhat irrelevant (e.g. `list`).  A `...` makes it somewhat unambiguous that there is more than what we see.  Leaving just the top-level doesn't give that impression.  I've had to explain what's going on a few times to people when sitting down with them.  I'm looking to make that conversation easier.  Alternatives welcome.

Moving the paren to the right matches how most people think of function calls.  I think that we should think in a lispy way internally but expose a non-lispy style externally whenever possible.

Agree with the above, I just thought we should truncate at a nice break, rather than mid-string. Here's what I have now (just for graphs, profiling will use whole version). Prints top-level function if there are no sub-tasks, otherwise prints top-level function with closed parens around `...`. I think this is a good mix of succinctness and clarity when there are sub-tasks.

![mydask](https://cloud.githubusercontent.com/assets/2783717/8752452/c24da130-2c79-11e5-84ce-9a5672a6a1e8.png)

This seems better than what was around before.  So +1 for now.

I wonder if we should rotate the graphs to be left-right or if we could use ellipses rather than circles to avoid size blowup

Thoughts on the lambda printing?

I think just printing `<lambda>` is the best we can do. The reason the profiler looks weird is because bokeh fails to properly escape html for the hovers, but this can be worked around by using a custom hover for now.


Resolving patch:
diff --git a/dask/diagnostics/profile.py b/dask/diagnostics/profile.py
--- a/dask/diagnostics/profile.py
+++ b/dask/diagnostics/profile.py
@@ -59,6 +59,7 @@ def __init__(self, get):
         """
         self._get = get
         self._results = {}
+        self._dsk = {}
 
     def _start_callback(self, key, dask, state):
         if key is not None:
@@ -77,6 +78,7 @@ def get(self, dsk, result, **kwargs):
         the dask."""
 
         self.clear()
+        self._dsk = dsk.copy()
         return self._get(dsk, result, start_callback=self._start_callback,
                          end_callback=self._end_callback, **kwargs)
 
@@ -95,8 +97,9 @@ def visualize(self, **kwargs):
         dask.diagnostics.profile_visualize.visualize
         """
         from .profile_visualize import visualize
-        return visualize(self.results(), **kwargs)
+        return visualize(self.results(), self._dsk, **kwargs)
 
     def clear(self):
         """Clear out old results from profiler"""
         self._results.clear()
+        self._dsk = {}
diff --git a/dask/diagnostics/profile_visualize.py b/dask/diagnostics/profile_visualize.py
--- a/dask/diagnostics/profile_visualize.py
+++ b/dask/diagnostics/profile_visualize.py
@@ -8,7 +8,52 @@
 from bokeh.palettes import brewer
 from bokeh.models import HoverTool
 
-from ..dot import name
+from ..dot import funcname
+from ..core import istask
+
+
+def pprint_task(task, keys):
+    """Return a nicely formatted string for a task.
+
+    Examples
+    --------
+    >>> from operator import add, mul
+    >>> dsk = {'a': 1,
+    ...        'b': 2,
+    ...        'c': (add, 'a', 'b'),
+    ...        'd': (add, (mul, 'a', 'b'), 'c'),
+    ...        'e': (sum, ['a', 'b', 5])}
+
+    >>> pprint_task(dsk['c'], dsk)
+    'add(_, _)'
+    >>> pprint_task(dsk['d'], dsk)
+    'add(mul(_, _), _)'
+    >>> pprint_task(dsk['e'], dsk)
+    'sum([_, _, 5])'
+    """
+    if istask(task):
+        func = task[0]
+        if hasattr(func, 'funcs'):
+            head = '('.join(funcname(f) for f in func.funcs)
+            tail = ')'*len(func.funcs)
+        else:
+            head = funcname(task[0])
+            tail = ')'
+        args = ', '.join(pprint_task(t, keys) for t in task[1:])
+        return '{0}({1}{2}'.format(head, args, tail)
+    elif isinstance(task, list):
+        args = ', '.join(pprint_task(t, keys) for t in task)
+        return '[{0}]'.format(args)
+    else:
+        try:
+            if task in keys:
+                return '_'
+        except TypeError:
+            pass
+        res = str(task)
+        if len(res) < 10:
+            return res
+        return '<{0}>'.format(type(task).__name__)
 
 
 def get_colors(palette, funcs):
@@ -36,7 +81,7 @@ def get_colors(palette, funcs):
     return [color_lookup[n] for n in funcs]
 
 
-def visualize(results, palette='GnBu', file_path="profile.html",
+def visualize(results, dsk, palette='GnBu', file_path="profile.html",
               show=True, **kwargs):
     """Visualize the results of profiling in a bokeh plot.
 
@@ -44,6 +89,8 @@ def visualize(results, palette='GnBu', file_path="profile.html",
     ----------
     results : sequence
         Output of profiler.results().
+    dsk : dict
+        The dask graph being profiled.
     palette : string, optional
         Name of the bokeh palette to use, must be key in bokeh.palettes.brewer.
     file_path : string, optional
@@ -75,6 +122,7 @@ def visualize(results, palette='GnBu', file_path="profile.html",
                     tools="hover,save,reset,resize,xwheel_zoom,xpan",
                     plot_width=800, plot_height=400)
     defaults.update(kwargs)
+
     p = bp.figure(y_range=[str(i) for i in range(len(id_lk))],
                   x_range=[0, right - left], **defaults)
 
@@ -82,12 +130,10 @@ def visualize(results, palette='GnBu', file_path="profile.html",
     data['width'] = width = [e - s for (s, e) in zip(starts, ends)]
     data['x'] = [w/2 + s - left for (w, s) in zip(width, starts)]
     data['y'] = [id_lk[i] + 1 for i in ids]
-    data['function'] = funcs = [name(i[0]) for i in tasks]
+    data['function'] = funcs = [pprint_task(i, dsk) for i in tasks]
     data['color'] = get_colors(palette, funcs)
+    data['key'] = [str(i) for i in keys]
 
-    # Bokeh barfs on quotes in hovers...
-    # https://github.com/bokeh/bokeh/issues/2094
-    data['key'] = [str(k).replace("'", "") for k in keys]
     source = bp.ColumnDataSource(data=data)
 
     p.rect(source=source, x='x', y='y', height=1, width='width',
@@ -98,9 +144,17 @@ def visualize(results, palette='GnBu', file_path="profile.html",
     p.yaxis.axis_label = "Worker ID"
     p.xaxis.axis_label = "Time (s)"
 
-    hover = p.select(type=HoverTool)
-    hover.tooltips = [("Key:", "@key"),
-                      ("Function:", "@function")]
+    hover = p.select(HoverTool)
+    hover.tooltips = """
+    <div>
+        <span style="font-size: 14px; font-weight: bold;">Key:</span>&nbsp;
+        <span style="font-size: 12px;">@key</span>
+    </div>
+    <div>
+        <span style="font-size: 14px; font-weight: bold;">Task:</span>&nbsp;
+        <span style="font-size: 12px;">@function</span>
+    </div>
+    """
     hover.point_policy = 'follow_mouse'
 
     if show:
diff --git a/dask/dot.py b/dask/dot.py
--- a/dask/dot.py
+++ b/dask/dot.py
@@ -7,13 +7,46 @@
 from .core import istask, get_dependencies, ishashable
 
 
-def label(func):
-    try:
-        while hasattr(func, 'func'):
-            func = func.func
-        return func.__name__
-    except AttributeError:
-        return 'func'
+def task_label(task):
+    """Label for a task on a dot graph.
+
+    Examples
+    --------
+    >>> from operator import add
+    >>> task_label((add, 1, 2))
+    'add'
+    >>> task_label((add, (add, 1, 2), 3))
+    'add(...)'
+    """
+    func = task[0]
+    if hasattr(func, 'funcs'):
+        if len(func.funcs) > 1:
+            return '{0}(...)'.format(funcname(func.funcs[0]))
+        else:
+            head = funcname(func.funcs[0])
+    else:
+        head = funcname(task[0])
+    if any(has_sub_tasks(i) for i in task[1:]):
+        return '{0}(...)'.format(head)
+    else:
+        return head
+
+
+def has_sub_tasks(task):
+    """Returns True if the task has sub tasks"""
+    if istask(task):
+        return True
+    elif isinstance(task, list):
+        return any(has_sub_tasks(i) for i in task)
+    else:
+        return False
+
+
+def funcname(func):
+    """Get the name of a function."""
+    while hasattr(func, 'func'):
+        func = func.func
+    return func.__name__
 
 
 def name(x):
@@ -41,11 +74,10 @@ def to_graphviz(dsk, data_attributes=None, function_attributes=None):
                    **data_attributes.get(k, {}))
 
         if istask(v):
-            func = v[0]
             func_name = name((k, 'function'))
             if func_name not in seen:
                 seen.add(func_name)
-                g.node(func_name, label=label(func), shape='circle',
+                g.node(func_name, label=task_label(v), shape='circle',
                        **function_attributes.get(k, {}))
             g.edge(func_name, k_name)
 


Test patch:
diff --git a/dask/diagnostics/tests/test_profiler.py b/dask/diagnostics/tests/test_profiler.py
--- a/dask/diagnostics/tests/test_profiler.py
+++ b/dask/diagnostics/tests/test_profiler.py
@@ -1,5 +1,5 @@
 from operator import add, mul
-from dask.diagnostics import Profiler, thread_prof, process_prof
+from dask.diagnostics import thread_prof
 import pytest
 try:
     import bokeh
@@ -26,9 +26,20 @@ def test_profiler():
     assert thread_prof.results() == []
 
 
+@pytest.mark.skipif("not bokeh")
+def test_pprint_task():
+    from dask.diagnostics.profile_visualize import pprint_task
+    keys = set(['a', 'b', 'c', 'd', 'e'])
+    assert pprint_task((add, 'a', 1), keys) == 'add(_, 1)'
+    assert pprint_task((add, (add, 'a', 1)), keys) == 'add(add(_, 1))'
+    res = 'sum([1, _, add(_, 1)])'
+    assert pprint_task((sum, [1, 'b', (add, 'a', 1)]), keys) == res
+    assert pprint_task((sum, (1, 2, 3, 4, 5, 6, 7)), keys) == 'sum(<tuple>)'
+
+
 @pytest.mark.skipif("not bokeh")
 def test_profiler_plot():
-    out = thread_prof.get(dsk, 'e')
+    thread_prof.get(dsk, 'e')
     # Run just to see that it doesn't error
     thread_prof.visualize(show=False)
     p = thread_prof.visualize(plot_width=500,
diff --git a/dask/tests/test_dot.py b/dask/tests/test_dot.py
--- a/dask/tests/test_dot.py
+++ b/dask/tests/test_dot.py
@@ -3,12 +3,11 @@
 import re
 from operator import add, neg
 
-import numpy as np
 import pytest
 
 pytest.importorskip("graphviz")
 
-from dask.dot import dot_graph, label, to_graphviz
+from dask.dot import dot_graph, task_label, to_graphviz
 
 # Since graphviz doesn't store a graph, we need to parse the output
 label_re = re.compile('.*\[label=(.*?) shape=.*\]')
@@ -26,8 +25,10 @@ def get_label(line):
        'f': (sum, ['a', 'e'])}
 
 
-def test_label():
-    assert label(partial(add, 1)) == 'add'
+def test_task_label():
+    assert task_label((partial(add, 1), 1)) == 'add'
+    assert task_label((add, 1)) == 'add'
+    assert task_label((add, (add, 1, 2))) == 'add(...)'
 
 
 def test_to_graphviz():