Repository: dask/dask

Issue:
Fold + List accumulator
This is odd behavior: 

``` python
import dask.bag as db

bag = db.from_sequence(range(0, 8, 1))

def append(accum, value):
    accum.append(value)
    return accum

def extend(accum, value):
    accum.extend(value)
    return accum

bag.fold(append, extend, initial=[]).compute()

## -- End pasted text --
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-32-b1f1bcf62207> in <module>()
     11     return accum
     12 
---> 13 bag.fold(append, extend, initial=[]).compute()

/Users/robstory/src/dask/dask/base.pyc in compute(self, **kwargs)
     25 
     26     def compute(self, **kwargs):
---> 27         return compute(self, **kwargs)[0]
     28 
     29     @classmethod

/Users/robstory/src/dask/dask/base.pyc in compute(*args, **kwargs)
     59                 for opt, val in groups.items()])
     60     keys = [arg._keys() for arg in args]
---> 61     results = get(dsk, keys, **kwargs)
     62     return tuple(a._finalize(a, r) for a, r in zip(args, results))

/Users/robstory/src/dask/dask/multiprocessing.pyc in get(dsk, keys, optimizations, num_workers, func_loads, func_dumps, **kwargs)
     54         # Run
     55         result = get_async(apply_async, len(pool._pool), dsk3, keys,
---> 56                            queue=queue, get_id=_process_get_id, **kwargs)
     57     finally:
     58         if cleanup:

/Users/robstory/src/dask/dask/async.pyc in get_async(apply_async, num_workers, dsk, result, cache, queue, get_id, raise_on_exception, rerun_exceptions_locally, callbacks, **kwargs)
    488                 "    dataset.compute(rerun_exceptions_locally=True)\n\n"
    489                 "The original exception and traceback follow below:\n\n"
--> 490                     + str(res) + "\n\nTraceback:\n" + tb)
    491         state['cache'][key] = res
    492         finish_task(dsk, key, state, results, keyorder.get)

AttributeError: Exception occurred in remote worker.

Something you've asked dask to compute raised an exception.
That exception and the traceback are copied below.
To use pdb, rerun the computation with the keyword argument
    dask.set_options(rerun_exceptions_locally=True)
    or
    dataset.compute(rerun_exceptions_locally=True)

The original exception and traceback follow below:

'generator' object has no attribute 'append'

Traceback:
  File "/Users/robstory/src/dask/dask/async.py", line 265, in execute_task
    result = _execute_task(task, data)
  File "/Users/robstory/src/dask/dask/async.py", line 248, in _execute_task
    return func(*args2)
  File "<ipython-input-32-b1f1bcf62207>", line 6, in append
    accum.append(value)
```

Why am I seeing a generator and not a list as my accumulator?



Discussion:
Oh, this is an interesting case.  

The dask schedulers interpret lists as `Iterators` this ends up being _very_ useful for a number of algorithms.  We usually end up quoting them into something like `(list, [])`.  

This clearly doesn't work for user generated stuff though; we can't expect users to be aware of this trait of the schedulers.  This again calls for a general `quote` function to sanitize inputs like this.  (@shoyer in particular has called for this a couple of times.)  I'll figure this out and apply it throughout the codebase hopefully sometime early next week.

In the mean time I'll special case this in fold.  PR forthcoming.


Resolving patch:
diff --git a/dask/bag/core.py b/dask/bag/core.py
--- a/dask/bag/core.py
+++ b/dask/bag/core.py
@@ -361,6 +361,8 @@ def fold(self, binop, combine=None, initial=no_default):
         """
         a = next(names)
         b = next(names)
+        if isinstance(initial, list):
+            initial = (list2, initial)
         if initial is not no_default:
             dsk = dict(((a, i), (reduce, binop, (self.name, i), initial))
                             for i in range(self.npartitions))


Test patch:
diff --git a/dask/bag/tests/test_bag.py b/dask/bag/tests/test_bag.py
--- a/dask/bag/tests/test_bag.py
+++ b/dask/bag/tests/test_bag.py
@@ -147,6 +147,10 @@ def binop(acc, x):
     d = db.from_sequence('hello')
     assert set(d.fold(lambda a, b: ''.join([a, b]), initial='').compute()) == set('hello')
 
+    e = db.from_sequence([[1], [2], [3]], npartitions=2)
+    with dask.set_options(get=get_sync):
+        assert set(e.fold(add, initial=[]).compute()) == set([1, 2, 3])
+
 
 def test_distinct():
     assert sorted(b.distinct()) == [0, 1, 2, 3, 4]