Repository: ipython/ipython

Issue:
use_dill does not work in Python 3.4
When executing a command like:

```
client[:].use_dill()
```

Every parallel operation outputs a long obscure pickle error:

```
def g(a,b):
    return a*b
parallel_client.client[:].apply_sync(g,2,4)
```

```
UnpicklingError                           Traceback (most recent call last)/home/zah/sourcecode/ipython/IPython/kernel/zmq/serialize.py in unpack_apply_message(bufs, g, copy)
    178         for i in range(2):
    179             bufs[i] = bufs[i].bytes
--> 180     f = uncan(pickle.loads(bufs.pop(0)), g)
    181     info = pickle.loads(bufs.pop(0))
    182     arg_bufs, kwarg_bufs = bufs[:info['narg_bufs']], bufs[info['narg_bufs']:]
/home/zah/anaconda3/lib/python3.4/site-packages/dill/dill.py in loads(str)
    158     """unpickle an object from a string"""
    159     file = StringIO(str)
--> 160     return load(file)
    161 
    162 # def dumpzs(obj, protocol=None):
/home/zah/anaconda3/lib/python3.4/site-packages/dill/dill.py in load(file)
    148     pik = Unpickler(file)
    149     pik._main_module = _main_module
--> 150     obj = pik.load()
    151     if type(obj).__module__ == _main_module.__name__: # point obj class to main
    152         try: obj.__class__ == getattr(pik._main_module, type(obj).__name__)
/home/zah/anaconda3/lib/python3.4/pickle.py in load(self)
   1034                     raise EOFError
   1035                 assert isinstance(key, bytes_types)
-> 1036                 dispatch[key[0]](self)
   1037         except _Stop as stopinst:
   1038             return stopinst.value
/home/zah/anaconda3/lib/python3.4/pickle.py in load_global(self)
   1319 
   1320     def load_global(self):
-> 1321         module = self.readline()[:-1].decode("utf-8")
   1322         name = self.readline()[:-1].decode("utf-8")
   1323         klass = self.find_class(module, name)
/home/zah/anaconda3/lib/python3.4/pickle.py in readline(self)
    245             if data[-1] != b'\n':
    246                 raise UnpicklingError(
--> 247                     "pickle exhausted before end of frame")
    248             return data
    249         else:
UnpicklingError: pickle exhausted before end of frame
```

The same works in 3.3.5



Discussion:
Sounds like a dill bug to me. Ping @mmckerns for any insight on any relevant changes in Python 3.4. What dill version?

@minrk: thanks for thinking of me for obscure pickling errors.

There are no changes that I've seen that impact python 3.4 vs 3.3.
However, I can tell by the line numbers above that it's not the bleeding edge version,
but it might be the most current stable version released 6/4, or it might be the version prior to that.  

Only recent bugs/fixes were a fix of stdin/stdout handling on 5/28 and one other bug I introduced to the version prior (0.2), released 5/17… that was patched 5/19.  Note both of the bugs (5/28) might be relevant… pipe --> StringIO, and the other (5/17) caused a conflict when import dill happened iff pickle was imported first.  Both are fixed in the most recent version of dill (0.2.1) released 6/4.

Aside from that, I could test most recent dill and ipython.parallel to see if it works, but I expect that you can also try that.

I'm not seeing an error for that function using my async_map ('Pool.amap') functions with most recent dill and python3.4.  I just ran it from a file, not the interpreter.  Pickling works different for different ways the thing get run… so any help on how it was run would be good too.

My not working dill version is 0.2.1, as installed from pip three days ago.

I'm not having issues with my own map functions and with `dill` in 3.4.   So can you send:
1) the version of `ipython` you are using,
2) and also exactly what you do to generate the error? 

It seems it might be a `dill` working in `ipython` issue… so since I'm a bit of an `ipython` novice, can you post the code you ran exactly so I can repeat your code and see if I get the error?

From zero:

```
conda create -n py34 python=3.4
source activate py34
#Version 2.1
conda install ipython-notebook
conda install pip
#Version 0.2.1
pip install dill
ipcluster start -n4
```

And then I get the error above.

Aha… so `ipcluster start -n4` that's what I was missing.  I see the error now, after `apply_sync` is called. I can confirm that it if I type what you did in the first entry, that I get the same error in 3.4, even if I'm using the bleeding edge version of `dill`.  However not when using python 3.3.  In both cases I'm using the latest macports `ipython`

```
In [1]: from IPython import parallel
In [2]: p = parallel.Client()
In [3]: p[:].use_dill()
Out[3]: <AsyncResult: use_dill>
In [4]: def g(a,c):
   ...:     return a*c
   ...: 
In [5]: p[:].apply_sync(g,2,4)
[0:apply]: 
---------------------------------------------------------------------------
UnpicklingError
```

I'll look into this.

So, this is a new function in `pickle` in python3.4.

```
def readline(self):
    if self.current_frame:
        data = self.current_frame.readline()
        if not data:
            self.current_frame = None
            return self.file_readline()
        if data[-1] != b'\n':
            raise UnpicklingError(
                "pickle exhausted before end of frame")
        return data
    else:
        return self.file_readline()
```

Since it's new in 3.4, and I haven't seen it throw an error before -- I don't know what it does yet. But I don't like it.

`__init__` in `pickle._Unpickler` looked like this in `3.3`...

```
self.readline = file.readline
```

but in 3.4 it's like this…

```
    self._unframer = _Unframer(self._file_read, self._file_readline)
    self.read = self._unframer.read
    self.readline = self._unframer.readline
```

where the `readline` method above is on the `_Unframer`.  and eventually it gets hooked up elsewhere:

```
self._file_readline = file.readline
```

Ok, so yuck.

@minrk: does `IPython` deep down do any selection of the pickling protocol?

python3.4 does some weird stuff, and `pickle` is still unstable… I found that it actually can pickle much much less stuff if you are using `HIGHEST_PROTOCOL`, which is `4`.   They added `DEFAULT_PROTOCOL`, which is new and is set to `3`… because  `4` is unstable.  So, `dill` uses `DEFAULT_PROTOCOL` unless you specify you want `HIGHEST_PROTOCOL`.  I don't ever do that, because  like I said `4` is unstable.

From `dill`:

```
def dump(obj, file, protocol=None, byref=False):
    """pickle an object to a file"""
    if protocol is None: protocol = DEFAULT_PROTOCOL
    pik = Pickler(file, protocol)
```

Then, magically, a lot of errors went away with pickling and python 3.4

I think IPython does hardcode HIGHEST_PROTOCOL. We should probably change this to DEFAULT_PROTOCOL, when defined. Thanks.

Hey, and I finally learned how to use `IPython.parallel`.  Yay me.  :)

@minrk: I just looked into this to see if it'd be easy to generate a pull request for you, and I can confirm that's definitely the root cause.  Everywhere you have `dumps(obj, -1)`, you should replace with `dumps(obj)`, and anywhere `dumps(obj, protocol)` is used, make sure `protocol=DEFAULT_PROTOCOL` (see below).  Surprisingly, it's all throughout the code, so I punted.  There's hardly ever a reason to use the "-1" since it's the default everywhere save for python 3.4.  Gotta love the new python.  

```
try:
    from pickle import DEFAULT_PROTOCOL
except ImportError:
    DEFAULT_PROTOCOL = pickle.HIGHEST_PROTOCOL
```

You'll probably have to do that too, since `DEFAULT_PROTOCOL` is "new" in 3.4.
This issue should be tedious, but easy to fix.  I'll pick it up it it's lingering too long.

The default in Python 2 is protocol 0. Only in Python 3 is the default version 3.

@takluyver: Ah… that's riiiiiight.  It's `dill` that always uses `HIGHEST_PROTOCOL` by default, save for python 3.4.  Still, it's the same fix as above… but with also replacing `-1` by `DEFAULT_PROTOCOL`

Yeah, I think it should be DEFAULT_PROTOCOL if defined, and either 2 or 3 if it isn't.

or HIGHEST

@minrk: `dill` uses DEFAULT_PROTOCOL if defined, and HIGHEST_PROTOCOL if not defined.  It's not the standard `pickle` choice, but it is the most common choice and a safe one to make -- unless one of the other applications specifically needs/uses a lower protocol.


Resolving patch:
diff --git a/IPython/kernel/zmq/serialize.py b/IPython/kernel/zmq/serialize.py
--- a/IPython/kernel/zmq/serialize.py
+++ b/IPython/kernel/zmq/serialize.py
@@ -1,19 +1,7 @@
-"""serialization utilities for apply messages
+"""serialization utilities for apply messages"""
 
-Authors:
-
-* Min RK
-"""
-#-----------------------------------------------------------------------------
-#  Copyright (C) 2010-2011  The IPython Development Team
-#
-#  Distributed under the terms of the BSD License.  The full license is in
-#  the file COPYING, distributed as part of this software.
-#-----------------------------------------------------------------------------
-
-#-----------------------------------------------------------------------------
-# Imports
-#-----------------------------------------------------------------------------
+# Copyright (c) IPython Development Team.
+# Distributed under the terms of the Modified BSD License.
 
 try:
     import cPickle
@@ -22,13 +10,12 @@
     cPickle = None
     import pickle
 
-
 # IPython imports
 from IPython.utils import py3compat
 from IPython.utils.data import flatten
 from IPython.utils.pickleutil import (
     can, uncan, can_sequence, uncan_sequence, CannedObject,
-    istype, sequence_types,
+    istype, sequence_types, PICKLE_PROTOCOL,
 )
 
 if py3compat.PY3:
@@ -99,7 +86,7 @@ def serialize_object(obj, buffer_threshold=MAX_BYTES, item_threshold=MAX_ITEMS):
         cobj = can(obj)
         buffers.extend(_extract_buffers(cobj, buffer_threshold))
 
-    buffers.insert(0, pickle.dumps(cobj,-1))
+    buffers.insert(0, pickle.dumps(cobj, PICKLE_PROTOCOL))
     return buffers
 
 def unserialize_object(buffers, g=None):
@@ -162,8 +149,8 @@ def pack_apply_message(f, args, kwargs, buffer_threshold=MAX_BYTES, item_thresho
     
     info = dict(nargs=len(args), narg_bufs=len(arg_bufs), kw_keys=kw_keys)
     
-    msg = [pickle.dumps(can(f),-1)]
-    msg.append(pickle.dumps(info, -1))
+    msg = [pickle.dumps(can(f), PICKLE_PROTOCOL)]
+    msg.append(pickle.dumps(info, PICKLE_PROTOCOL))
     msg.extend(arg_bufs)
     msg.extend(kwarg_bufs)
     
diff --git a/IPython/kernel/zmq/session.py b/IPython/kernel/zmq/session.py
--- a/IPython/kernel/zmq/session.py
+++ b/IPython/kernel/zmq/session.py
@@ -43,6 +43,7 @@
                                         DottedObjectName, CUnicode, Dict, Integer,
                                         TraitError,
 )
+from IPython.utils.pickleutil import PICKLE_PROTOCOL
 from IPython.kernel.adapter import adapt
 from IPython.kernel.zmq.serialize import MAX_ITEMS, MAX_BYTES
 
@@ -76,7 +77,7 @@ def squash_unicode(obj):
 )
 json_unpacker = lambda s: jsonapi.loads(s)
 
-pickle_packer = lambda o: pickle.dumps(squash_dates(o),-1)
+pickle_packer = lambda o: pickle.dumps(squash_dates(o), PICKLE_PROTOCOL)
 pickle_unpacker = pickle.loads
 
 default_packer = json_packer
diff --git a/IPython/utils/pickleutil.py b/IPython/utils/pickleutil.py
--- a/IPython/utils/pickleutil.py
+++ b/IPython/utils/pickleutil.py
@@ -28,6 +28,11 @@
     from types import ClassType
     class_type = (type, ClassType)
 
+try:
+    PICKLE_PROTOCOL = pickle.DEFAULT_PROTOCOL
+except AttributeError:
+    PICKLE_PROTOCOL = pickle.HIGHEST_PROTOCOL
+
 def _get_cell_type(a=None):
     """the type of a closure cell doesn't seem to be importable,
     so just create one
@@ -244,7 +249,7 @@ def __init__(self, obj):
             self.pickled = True
         if self.pickled:
             # just pickle it
-            self.buffers = [pickle.dumps(obj, -1)]
+            self.buffers = [pickle.dumps(obj, PICKLE_PROTOCOL)]
         else:
             # ensure contiguous
             obj = ascontiguousarray(obj, dtype=None)


Test patch: