Repository: dask/dask

Issue:
Inheritance of Purity
(As discussed here: http://stackoverflow.com/questions/42773134/inheritance-of-purity/42773643)

I have a question about inheritance of function purity. For example, consider this case:

```python
In [1]: from dask import delayed

In [2]: myArr = delayed(np.ones, pure=True)((10,10))

In [3]: myArr
Out[3]: Delayed('ones-850ff41a84b309775d01d5f3e5c4d1c4')

In [4]: myArr = delayed(np.ones, pure=True)((10,10))

In [5]: myArr
Out[5]: Delayed('ones-850ff41a84b309775d01d5f3e5c4d1c4')

In [6]: myArr.shape
Out[6]: Delayed('getattr-de14f312-2605-4fc4-a419-c376520f73b4')

In [7]: myArr.shape
Out[7]: Delayed('getattr-9e547a62-0bc6-43e5-8d3c-cac532b73511')

In [8]: delayed(getattr, pure=True)(myArr, 'shape')
Out[8]: Delayed('getattr-5224be928bad33c9778d4f96d610cc37')

In [9]: delayed(getattr, pure=True)(myArr, 'shape')
Out[9]: Delayed('getattr-5224be928bad33c9778d4f96d610cc37')

```

I would expect line [6] and [7] to yield the same key, since they are accessing an attribute from a delayed instance declared pure. However, it is not the case, and I must explicitly declare the `getattr` method to be pure for this to work, as seen in lines [8] and [9].

As discussed in the stackoverflow link, this seems to be purely an issue of `dask` semantics. I would be inclined to voting for purity to be inherited. But I am a new user, so I would appreciate feedback/comments on this. Thanks!

PS: The eventual use case is for dask distributed. It seems that the way purity is treated in this case may possibly be different than the case listed here.


Discussion:
`delayed(getattr,pure=True)(myArr, 'shape',pure=True)` looks like one more `pure=True` than necessary -- are both of those really necessary?
oops yes, that was a typo, it still ran since it's treated as a kwarg to a delayed object. I corrected it, thanks!
For the record, `getitem` seems to be forced as `pure=True`, as opposed to `getattr`:

```python
In [6]: myArr = delayed(np.ones, pure=False)((10,10))

In [7]: myArr[0]
Out[7]: Delayed('getitem-f7b0bc2869f1fe6c1221edc2aaac0a73')

In [8]: myArr[0]
Out[8]: Delayed('getitem-f7b0bc2869f1fe6c1221edc2aaac0a73')
```

Would it make sense instead of choosing purity from an absolute standpoint to choose it from a relative inherited standpoint? I don't think `getitem` will always necessary yield something pure. (For example, could be an object with a simulated set of random images fed into some image processing algorithm that runs on image sequences, in a loop.) This may be opening a can of worms. Any discussion/feedback appreciated!
Just to be clear on terms here, a delayed object does not currently know if it is "pure" or not.  Only delayed functions can be labeled as pure.  I suspect that the confusion here arises because we use the function `delayed` both to mark a value as part of a graph and to decorate a function.

Even if we kept the label around it's not clear that the attributes of "pure" objects would themselves be pure.
I'd be fine with the following:

- There's no concept of a "pure" object. Nested attribute accesses, function calls, etc... make this too hard to reason about. This should be clarified in the docs as needed.
- Attribute access is always pure. We already assume that all "magic" methods on delayed objects are pure, so we shouldn't make `__getattr__` the exception (which was unintentional).
- Perhaps error if `delayed` is called on a non-callable object with `pure=True` to better indicate that `pure=True` only indicates that calling the resulting object is a pure operation.

Thoughts?
`delayed(obj, pure=True)` is currently used as a signal that we should take the trouble to hash the object down to get the key.
Ah, ok, not the last one then.

Resolving patch:
diff --git a/dask/delayed.py b/dask/delayed.py
--- a/dask/delayed.py
+++ b/dask/delayed.py
@@ -240,9 +240,29 @@ def delayed(obj, name=None, pure=False, nout=None, traverse=True):
     >>> res.compute()  # doctest: +SKIP
     AttributeError("'list' object has no attribute 'not_a_real_method'")
 
-    Methods are assumed to be impure by default, meaning that subsequent calls
-    may return different results. To assume purity, set `pure=True`. This
-    allows sharing of any intermediate values.
+    "Magic" methods (e.g. operators and attribute access) are assumed to be
+    pure, meaning that subsequent calls must return the same results.  This is
+    not overrideable. To invoke an impure attribute or operator, you'd need to
+    use it in a delayed function with ``pure=False``.
+
+    >>> class Incrementer(object):
+    ...     def __init__(self):
+    ...         self._n = 0
+    ...     @property
+    ...     def n(self):
+    ...         self._n += 1
+    ...         return self._n
+    ...
+    >>> x = delayed(Incrementer())
+    >>> x.n.key == x.n.key
+    True
+    >>> get_n = delayed(lambda x: x.n, pure=False)
+    >>> get_n(x).key == get_n(x).key
+    False
+
+    In contrast, methods are assumed to be impure by default, meaning that
+    subsequent calls may return different results. To assume purity, set
+    `pure=True`. This allows sharing of any intermediate values.
 
     >>> a.count(2, pure=True).key == a.count(2, pure=True).key
     True
@@ -252,7 +272,6 @@ def delayed(obj, name=None, pure=False, nout=None, traverse=True):
 
     >>> a.count(2, dask_key_name="count_2")
     Delayed("count_2")
-
     """
     if isinstance(obj, Delayed):
         return obj
@@ -346,7 +365,7 @@ def __dir__(self):
     def __getattr__(self, attr):
         if attr.startswith('_'):
             raise AttributeError("Attribute {0} not found".format(attr))
-        return DelayedAttr(self, attr, 'getattr-%s' % tokenize(self, attr))
+        return DelayedAttr(self, attr)
 
     def __setattr__(self, attr, val):
         if attr in self.__slots__:
@@ -436,10 +455,10 @@ def __call__(self, *args, **kwargs):
 class DelayedAttr(Delayed):
     __slots__ = ('_obj', '_attr', '_key')
 
-    def __init__(self, obj, attr, key):
+    def __init__(self, obj, attr):
         self._obj = obj
         self._attr = attr
-        self._key = key
+        self._key = 'getattr-%s' % tokenize(obj, attr, pure=True)
 
     @property
     def dask(self):


Test patch:
diff --git a/dask/tests/test_delayed.py b/dask/tests/test_delayed.py
--- a/dask/tests/test_delayed.py
+++ b/dask/tests/test_delayed.py
@@ -69,6 +69,7 @@ def test_methods():
 
 def test_attributes():
     a = delayed(2 + 1j)
+    assert a.real._key == a.real._key
     assert a.real.compute() == 2
     assert a.imag.compute() == 1
     assert (a.real + a.imag).compute() == 3