Repository: dask/dask

Issue:
calling delayed functions with lists is slow
The following is relatively slow:

```
myfun = lambda *args, **kwargs: None
%time d = [dask.delayed(myfun)(list(range(10000))) for i in range(100)]
```

> CPU times: user 11.2 s, sys: 88 ms, total: 11.3 s
> Wall time: 11.3 s

Since generating 100 lists of length 10k only takes about 50ms, the majority of the CPU time seems to be due to dask.delayed. What is dask.delayed doing behind the scenes?

Edit:
```
dask.__version__
> u'0.12.0'
```


Discussion:
It's checking if any element of the list is another delayed object, or a list/dict/set/tuple that might contain a delayed object.
Okay, so I probably should avoid passing these types to `delayed` in general. 

Can I bypass the search that `dask.delayed` performs? Currently I try to hide all lists from dask delayed, e.g. using function enclosures, which feels a bit like a work around. 
Relevant code is here: https://github.com/dask/dask/blob/bef55f24cba8687d93a0d08b1ac76cd409d70283/dask/delayed.py#L31-L88

```python
def to_task_dasks(expr):
    """Normalize a python object and extract all sub-dasks.
    - Replace ``Delayed`` with their keys
    - Convert literals to things the schedulers can handle
    - Extract dasks from all enclosed values
    Parameters
    ----------
    expr : object
        The object to be normalized. This function knows how to handle
        ``Delayed``s, as well as most builtin python types.
    Returns
    -------
    task : normalized task to be run
    dasks : list of dasks that form the dag for this task
    Examples
    --------
    >>> a = delayed(1, 'a')
    >>> b = delayed(2, 'b')
    >>> task, dasks = to_task_dasks([a, b, 3])
    >>> task # doctest: +SKIP
    ['a', 'b', 3]
    >>> dasks # doctest: +SKIP
    [{'a': 1}, {'b': 2}]
    >>> task, dasks = to_task_dasks({a: 1, b: 2})
    >>> task # doctest: +SKIP
    (dict, [['a', 1], ['b', 2]])
    >>> dasks # doctest: +SKIP
    [{'a': 1}, {'b': 2}]
    """
    if isinstance(expr, Delayed):
        return expr.key, expr._dasks
    if isinstance(expr, base.Base):
        name = 'finalize-' + tokenize(expr, pure=True)
        keys = expr._keys()
        dsk = expr._optimize(expr.dask, keys)
        dsk[name] = (expr._finalize, (concrete, keys))
        return name, [dsk]
    if isinstance(expr, tuple) and type(expr) != tuple:
        return expr, []
    if isinstance(expr, (Iterator, list, tuple, set)):
        args, dasks = unzip((to_task_dasks(e) for e in expr), 2)
        args = list(args)
        dasks = flat_unique(dasks)
        # Ensure output type matches input type
        if isinstance(expr, (tuple, set)):
            return (type(expr), args), dasks
        else:
            return args, dasks
    if isinstance(expr, dict):
        args, dasks = to_task_dasks([[k, v] for k, v in expr.items()])
        return (dict, args), dasks
    return expr, []
```

Delayed dives within standard Python collections (tuple, list, dict, etc.)  Making it possible to avoid searching seems reasonable but isn't currently implemented (to my knowledge).  Alternatively you could also use a container that is opaque to delayed like a NumPy array.  

*Ideally* you don't call `delayed` on data, but rather on the function that would produce the data.  This isn't always posible though.
Thanks for pointing me to that function. 

If I pass a list of integers to that function, it appears to me that we get a list of empty dasks of the same length. Probably, they have to be merged later which also takes some more time. Would it make sense to add a test before the recursive call of `to_task_dasks`, so that we only enter the recursion, if there is at least one "interesting" object (another list, base.Base object, ...) in the list?
Yeah, I can imagine that some common case filtering might help out considerably. 

Here is a profile of your example:

```python
In [4]: %prun d = [dask.delayed(myfun)(list(range(10000))) for i in range(100)]
         10006976 function calls (9006874 primitive calls) in 4.728 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
  1000200    1.464    0.000    1.977    0.000 abc.py:178(__instancecheck__)
1000200/200    1.012    0.000    4.700    0.023 delayed.py:31(to_task_dasks)
  5001500    0.943    0.000    2.920    0.000 {built-in method builtins.isinstance}
  2000408    0.513    0.000    0.513    0.000 _weakrefset.py:70(__contains__)
  200/100    0.496    0.002    4.699    0.047 delayed.py:23(unzip)
  1000100    0.229    0.000    4.142    0.000 delayed.py:77(<genexpr>)
      200    0.043    0.000    0.043    0.000 itertoolz.py:193(unique)
        1    0.016    0.016    4.728    4.728 <string>:1(<listcomp>)
      100    0.002    0.000    4.705    0.047 delayed.py:391(call_function)
      200    0.001    0.000    0.001    0.000 {built-in method posix.urandom}
      200    0.001    0.000    0.002    0.000 uuid.py:106(__init__)
      200    0.001    0.000    0.044    0.000 delayed.py:18(flat_unique)
      200    0.001    0.000    0.001    0.000 uuid.py:226(__str__)
      700    0.001    0.000    0.001    0.000 delayed.py:348(__setattr__)
      200    0.001    0.000    0.006    0.000 delayed.py:91(tokenize)
```
To figure out to what extend filtering might help, I did the following experiment:
```
args, dasks = unzip((to_task_dasks(e) for e in expr \
                             if isinstance(e, (Delayed, base.Base, Iterator, list, tuple, set, dict))), 2)
```
This gives ~50% speed increase, but not more. The remaining time seems to be on account of the 100*1000 calls of isinstance, which are probably inevitable if delayed is supposed to check every element in the list. So to get a significant speedup, I think the user needs to provide additional information.

Here an idea how the user interface could be extended:
We introduce the class `opaque`, which serves  as a simple wrapper around an arbitrary object. Delayed guarantees not to look into opaque objects. When compute is called, the underlying function is called with the object stored within `opaque`. So the original example would become:

```
from dask.delayed import opaque as o
myfun = lambda *args, **kwargs: None
%time d = [dask.delayed(myfun)(o(mylist)) for i in range(100)]
```
What do you think? Do you see an opportunity for a significant speed up that does not require to change the user interface?
Instead of adding another class/function, I suggest adding a new keyword to `delayed`, and using that to pre-wrap large objects. Perhaps call it `deep` or `traverse`. The default would be `True`, which is the current behavior, but you could set it to `False` to tell dask not to do a deep crawl of objects that are being wrapped. Then your application would be:

```python
d = delayed(myfun)(delayed(mylist, deep=False))
```
I lightly prefer traverse over deep.

We might want to try using type checks in some cases:

```python
typ = type(x)

# if isinstance(x, Foo):
if typ is Foo:  # this is faster 
```

I think that this might make sense for `Delayed`, `list`, `tuple`, and `set`.   @jcrist, any thoughts or warnings here?

Even if we implement deep/traverse it would still be good to do performance optimizations in the True case.  The naive users won't find the deep/traverse option.
Using `type(x) is foo` checks sound fine to me. We do that in several places elsewhere as well.
The advantage of an additional opaque object might be, that the user can easily mix lists of delayed objects with data lists, something like: 

`delayed(myfun)(o(my_data_list), my_delayed_list)`

How could something similar be expressed with the API that provides a `deep` / `traverse` keyword?


Once a python object is wrapped in a `delayed`, it's no longer traversed (and is effectively opaque). So the options are equivalent.

```python
my_list = [1, 2, 3]
# Looks at each element in the list to see if it's a dask object,
# before returning an instance of `Delayed`
delayed_list = delayed(my_list)
# Doesn't do the traversal, and just returns an instance of `Delayed`
delayed_list_2 = delayed(my_list, traverse=False)
# In the call here `my_list` is traversed (since it's a python object).
# Neither `delayed_list` or `delayed_list_2` are traversed, since
# they're already wrapped in `delayed`
delayed(myfun)(my_list, delayed_list, delayed_list_2)
```

Does that make sense?
Okay, I see. So I am completely fine with the approach you suggested.
So, whats the plan ... is someone already implementing the traverse keyword? Should I give it a try?

Resolving patch:
diff --git a/dask/bag/core.py b/dask/bag/core.py
--- a/dask/bag/core.py
+++ b/dask/bag/core.py
@@ -635,7 +635,6 @@ def fold(self, binop, combine=None, initial=no_default, split_every=None):
         Bag.foldby
         """
         combine = combine or binop
-        initial = quote(initial)
         if initial is not no_default:
             return self.reduction(curry(_reduce, binop, initial=initial),
                                   curry(_reduce, combine),
diff --git a/dask/core.py b/dask/core.py
--- a/dask/core.py
+++ b/dask/core.py
@@ -423,6 +423,23 @@ def isdag(d, keys):
     return not getcycle(d, keys)
 
 
+class literal(object):
+    """A small serializable object to wrap literal values without copying"""
+    __slots__ = ('data',)
+
+    def __init__(self, data):
+        self.data = data
+
+    def __repr__(self):
+        return 'literal<type=%s>' % type(self.data).__name__
+
+    def __reduce__(self):
+        return (literal, (self.data,))
+
+    def __call__(self):
+        return self.data
+
+
 def quote(x):
     """ Ensure that this value remains this value in a dask graph
 
@@ -430,8 +447,8 @@ def quote(x):
     ensure that our data is not interpreted but remains literal.
 
     >>> quote((add, 1, 2))  # doctest: +SKIP
-    (tuple, [add, 1, 2])
+    (literal<type=tuple>,)
     """
-    if istask(x):
-        return (tuple, list(map(quote, x)))
+    if istask(x) or type(x) is list:
+        return (literal(x),)
     return x
diff --git a/dask/delayed.py b/dask/delayed.py
--- a/dask/delayed.py
+++ b/dask/delayed.py
@@ -7,6 +7,7 @@
 
 from toolz import merge, unique, curry, first
 
+from .core import quote
 from .utils import concrete, funcname, methodcaller
 from . import base
 from .compatibility import apply
@@ -107,7 +108,7 @@ def tokenize(*args, **kwargs):
 
 
 @curry
-def delayed(obj, name=None, pure=False, nout=None):
+def delayed(obj, name=None, pure=False, nout=None, traverse=True):
     """Wraps a function or object to produce a ``Delayed``.
 
     ``Delayed`` objects act as proxies for the object they wrap, but all
@@ -128,9 +129,14 @@ def delayed(obj, name=None, pure=False, nout=None):
         The number of outputs returned from calling the resulting ``Delayed``
         object. If provided, the ``Delayed`` output of the call can be iterated
         into ``nout`` objects, allowing for unpacking of results. By default
-        iteration over ``Delayed`` objects will error.
-        Note, that ``nout=1`` expects ``obj``, to return a tuple of length 1,
-        and consequently for `nout=0``, ``obj`` should return an empty tuple.
+        iteration over ``Delayed`` objects will error. Note, that ``nout=1``
+        expects ``obj``, to return a tuple of length 1, and consequently for
+        `nout=0``, ``obj`` should return an empty tuple.
+    traverse : bool, optional
+        By default dask traverses builtin python collections looking for dask
+        objects passed to ``delayed``. For large collections this can be
+        expensive. If ``obj`` doesn't contain any dask objects, set
+        ``traverse=False`` to avoid doing this traversal.
 
     Examples
     --------
@@ -248,9 +254,13 @@ def delayed(obj, name=None, pure=False, nout=None):
     if isinstance(obj, Delayed):
         return obj
 
-    task, dasks = to_task_dasks(obj)
+    if isinstance(obj, base.Base) or traverse:
+        task, dasks = to_task_dasks(obj)
+    else:
+        task = quote(obj)
+        dasks = []
 
-    if not dasks:
+    if task is obj:
         if not (nout is None or (type(nout) is int and nout >= 0)):
             raise ValueError("nout must be None or a positive integer,"
                              " got %s" % nout)


Test patch:
diff --git a/dask/tests/test_core.py b/dask/tests/test_core.py
--- a/dask/tests/test_core.py
+++ b/dask/tests/test_core.py
@@ -1,11 +1,12 @@
 from collections import namedtuple
 
 import pytest
+import pickle
 
 from dask.utils_test import GetFunctionTestMixin, inc, add
 from dask import core
 from dask.core import (istask, get_dependencies, get_deps, flatten, subs,
-                       preorder_traversal, quote, has_tasks)
+                       preorder_traversal, literal, quote, has_tasks)
 
 
 def contains(a, b):
@@ -226,3 +227,8 @@ def test_quote():
 
     for l in literals:
         assert core.get({'x': quote(l)}, 'x') == l
+
+
+def test_literal_serializable():
+    l = literal((add, 1, 2))
+    assert pickle.loads(pickle.dumps(l)).data == (add, 1, 2)
diff --git a/dask/tests/test_delayed.py b/dask/tests/test_delayed.py
--- a/dask/tests/test_delayed.py
+++ b/dask/tests/test_delayed.py
@@ -176,6 +176,41 @@ def f(seq):
     assert c.compute() == 3
 
 
+def test_traverse_false():
+    # Create a list with a dask value, and test that it's not computed
+    def fail(*args):
+        raise ValueError("shouldn't have computed")
+
+    a = delayed(fail)()
+
+    # list
+    x = [a, 1, 2, 3]
+    res = delayed(x, traverse=False).compute()
+    assert len(res) == 4
+    assert res[0] is a
+    assert res[1:] == x[1:]
+
+    # tuple that looks like a task
+    x = (fail, a, (fail, a))
+    res = delayed(x, traverse=False).compute()
+    assert isinstance(res, tuple)
+    assert res[0] == fail
+    assert res[1] is a
+
+    # list containing task-like-things
+    x = [1, (fail, a), a]
+    res = delayed(x, traverse=False).compute()
+    assert isinstance(res, list)
+    assert res[0] == 1
+    assert res[1][0] == fail and res[1][1] is a
+    assert res[2] is a
+
+    # traverse=False still hits top level
+    b = delayed(1)
+    x = delayed(b, traverse=False)
+    assert x.compute() == 1
+
+
 def test_pure():
     v1 = delayed(add, pure=True)(1, 2)
     v2 = delayed(add, pure=True)(1, 2)