Repository: dask/dask

Issue:
Should dask.delayed cull for optimization?
Currently `dask.Delayed.__dask_optimize__` is `dont_optimize`.  Do we want to add something simple, like `cull`?




Discussion:
I had to work around this here: https://github.com/dask/distributed/pull/2100
It looks like the local scheduler culls by default and the distributed scheduler doesn't.  
Had the impression this was expensive hence why it was not optimized.

Interesting to hear the Distributed scheduler doesn't cull. Though not surprising in retrospect.
Yes, it is a bit of a cost.  Typically we suffer this cost in the
collection's optimizations.  That's not the case for delayed though, which
is the only collection that doesn't cull by default

On Sat, Jul 14, 2018 at 6:17 PM, jakirkham <notifications@github.com> wrote:

> Had the impression this was expensive hence why it was not optimized.
>
> Interesting to hear the Distributed scheduler doesn't cull. Though not
> surprising in retrospect.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/3735#issuecomment-405055837>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AASszAgpgBrzL93mgRlx-Xay-cOS8wkmks5uGnwegaJpZM4VGVpB>
> .
>

If you have a chance to comment @jcrist, it would be good to hear your thoughts on optimizing delayed collections.
Apologies for the delay here (I'm on vacation this week). If other collections cull, I don't see a reason why delayed shouldn't. However, since the local scheduler culls as well, this leads to work being done twice - it might be good to either remove the culling in the local scheduler (may break/slow down people using custom graphs), or add it to the distributed scheduler and remove it from the collections.
It's useful to cull before doing other optimizations, so keeping it in the
standard optimization suite for array/dataframe/bag is definitely a win.

I'm somewhat inclined to remove culling from the local scheduler

On Wed, Jul 18, 2018 at 10:01 AM, Jim Crist <notifications@github.com>
wrote:

> Apologies for the delay here (I'm on vacation this week). If other
> collections cull, I don't see a reason why delayed shouldn't. However,
> since the local scheduler culls as well, this leads to work being done
> twice - it might be good to either remove the culling in the local
> scheduler (may break/slow down people using custom graphs), or add it to
> the distributed scheduler and remove it from the collections.
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/dask/dask/issues/3735#issuecomment-405941736>, or mute
> the thread
> <https://github.com/notifications/unsubscribe-auth/AASszAqxG2xCNrRR4LSjgSz-RV3lqinLks5uHz_BgaJpZM4VGVpB>
> .
>

Agreed. For people using custom graphs, they know enough that they can add `cull`. We should just make sure to put this in the changelog so that people are aware they need to take this action.

Resolving patch:
diff --git a/dask/delayed.py b/dask/delayed.py
--- a/dask/delayed.py
+++ b/dask/delayed.py
@@ -16,6 +16,7 @@
 from .compatibility import apply
 from .core import quote
 from .context import globalmethod
+from .optimization import cull
 from .utils import funcname, methodcaller, OperatorMethodMixin
 from . import sharedict
 
@@ -339,6 +340,11 @@ def _inner(self, other):
     return _inner
 
 
+def optimize(dsk, keys, **kwargs):
+    dsk2, _ = cull(dsk, keys)
+    return dsk2
+
+
 def rebuild(dsk, key, length):
     return Delayed(key, dsk, length)
 
@@ -367,7 +373,7 @@ def __dask_tokenize__(self):
         return self.key
 
     __dask_scheduler__ = staticmethod(threaded.get)
-    __dask_optimize__ = globalmethod(dont_optimize, key='delayed_optimize')
+    __dask_optimize__ = globalmethod(optimize, key='delayed_optimize')
 
     def __dask_postcompute__(self):
         return single_key, ()
diff --git a/dask/local.py b/dask/local.py
--- a/dask/local.py
+++ b/dask/local.py
@@ -115,7 +115,6 @@
 from . import config
 from .order import order
 from .callbacks import unpack_callbacks, local_callbacks
-from .optimization import cull
 from .utils_test import add, inc  # noqa: F401
 
 
@@ -454,8 +453,6 @@ def get_async(apply_async, num_workers, dsk, result, cache=None,
                     cb[0](dsk)
                 started_cbs.append(cb)
 
-            dsk, dependencies = cull(dsk, list(results))
-
             keyorder = order(dsk)
 
             state = start_state_from_dask(dsk, cache=cache, sortkey=keyorder.get)


Test patch:
diff --git a/dask/tests/test_delayed.py b/dask/tests/test_delayed.py
--- a/dask/tests/test_delayed.py
+++ b/dask/tests/test_delayed.py
@@ -164,6 +164,15 @@ def test_common_subexpressions():
     assert len(res.dask) == 3
 
 
+def test_delayed_optimize():
+    x = Delayed('b', {'a': 1,
+                      'b': (inc, 'a'),
+                      'c': (inc, 'b')})
+    (x2,) = dask.optimize(x)
+    # Delayed's __dask_optimize__ culls out 'c'
+    assert sorted(x2.dask.keys()) == ['a', 'b']
+
+
 def test_lists():
     a = delayed(1)
     b = delayed(2)