Repository: ipython/ipython

Issue:
Serialize methods and closures
IPython+Pickle can't serialize class methods.

``` Python
>>> from IPython.parallel import Client
>>> view = Client()[:]
>>> view.map(str.split, ['hello world', 'foo bar'])

PicklingError: Can't pickle <type 'method_descriptor'>: attribute lookup __builtin__.method_descriptor failed
```

IPython+Pickle can't serialze closures

``` Python
/home/mrocklin/Software/anaconda/lib/python2.7/site-packages/IPython/utils/codeutil.pyc in reduce_code(co)
     32 def reduce_code(co):
     33     if co.co_freevars or co.co_cellvars:
---> 34         raise ValueError("Sorry, cannot pickle code objects with closures")
     35     args =  [co.co_argcount, co.co_nlocals, co.co_stacksize,
     36             co.co_flags, co.co_code, co.co_consts, co.co_names,

ValueError: Sorry, cannot pickle code objects with closures
```

The `dill` project (CC @mmckerns) _can_ serialize code in both of these cases.  Is there a way to leverage `dill` here?

One somewhat hacky option would be to fall back on `pickle` rather than `cPickle` in case of failure.  Dill is able to register its serialization routines with the pure Python implementation.

``` Python
In [1]: import pickle
In [2]: pickle.dumps(str.split)
TypeError: can't pickle method_descriptor objects

In [3]: import dill
In [4]: pickle.dumps(str.split)  # Note: still using pickle interface
Out[4]: 'cdill.dill\n_getattr\np0\n(cdill.dill\n_load_type\np1\n(S\'StringType\'\np2\ntp3\nRp4\nS\'split\'\np5\nS"<method \'split\' of \'str\' objects>"\np6\ntp7\nRp8\n.'
```



Discussion:
Yes, there is. Since IPython (mostly) just uses pickle, `import dill` gets you 99% there. I've illustrated this [here](http://nbviewer.ipython.org/5241793).  I've been meaning to add a public `use_dill` API that will enable this cleanly and clearly.

Great!  Thanks for the pointer @minrk .  A `use_dill` routine in some utils file would be appreciated.  While the process outlined in your notebook is straightforward it's not the sort of thing I'll want to remember each time I have to do it.

I'll assign this issue to me, so that I remember to make that use_dill entry point


Resolving patch:
diff --git a/IPython/parallel/client/view.py b/IPython/parallel/client/view.py
--- a/IPython/parallel/client/view.py
+++ b/IPython/parallel/client/view.py
@@ -25,6 +25,7 @@
 import zmq
 
 from IPython.testing.skipdoctest import skip_doctest
+from IPython.utils import pickleutil
 from IPython.utils.traitlets import (
     HasTraits, Any, Bool, List, Dict, Set, Instance, CFloat, Integer
 )
@@ -511,6 +512,16 @@ def view_import(name, globals={}, locals={}, fromlist=[], level=0):
         for r in results:
             # raise possible remote ImportErrors here
             r.get()
+    
+    def use_dill(self):
+        """Expand serialization support with dill
+        
+        adds support for closures, etc.
+        
+        This calls IPython.utils.pickleutil.use_dill() here and on each engine.
+        """
+        pickleutil.use_dill()
+        return self.apply(pickleutil.use_dill)
 
 
     @sync_results
diff --git a/IPython/utils/pickleutil.py b/IPython/utils/pickleutil.py
--- a/IPython/utils/pickleutil.py
+++ b/IPython/utils/pickleutil.py
@@ -39,6 +39,36 @@
     from types import ClassType
     class_type = (type, ClassType)
 
+#-------------------------------------------------------------------------------
+# Functions
+#-------------------------------------------------------------------------------
+
+
+def use_dill():
+    """use dill to expand serialization support
+    
+    adds support for object methods and closures to serialization.
+    """
+    # import dill causes most of the magic
+    import dill
+    
+    # dill doesn't work with cPickle,
+    # tell the two relevant modules to use plain pickle
+    
+    global pickle
+    pickle = dill
+
+    try:
+        from IPython.kernel.zmq import serialize
+    except ImportError:
+        pass
+    else:
+        serialize.pickle = dill
+    
+    # disable special function handling, let dill take care of it
+    can_map.pop(FunctionType, None)
+
+
 #-------------------------------------------------------------------------------
 # Classes
 #-------------------------------------------------------------------------------


Test patch: