Repository: ipython/ipython

Issue:
IPython.parallel issue with pushing pandas TimeSeries
A pandas.TimeSeries is cast to a numpy.ndarray when pushed using IPython.parallel.

Here is an example:

```
In [1]: from IPython import parallel

In [2]: dview = parallel.Client()[:]

In [3]: with dview.sync_imports(): import pandas
importing pandas on engine(s)

In [4]: a = pandas.TimeSeries([1,2,3],index=[1,2,3])

In [5]: a.__class__
pandas.core.series.Series

In [6]: dview.push({'b':a})
<AsyncResult: _push>

In [7]: ret = dview.apply_async(lambda x: b.__class__, range(0,1))

In [8]: ret.result
[numpy.ndarray,
 numpy.ndarray]
```

This seems to be a result of dview.push handling numpy arrays differently: http://ipython.org/ipython-doc/dev/parallel/parallel_details.html
and pandas.TimeSeries using numpy arrays:

```
In [9]: cPickle.dumps(a)
"cnumpy.core.multiarray\n_reconstruct\np1\n(cpandas.core.series\nSeries\np2\n(I0\ntS'b'\ntRp3\n((I1\n(I3\ntcnumpy\ndtype\np4\n(S'i8'\nI0\nI1\ntRp5\n(I3\nS'<'\nNNNI-1\nI-1\nI0\ntbI00\nS'\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x03\\x00\\x00\\x00\\x00\\x00\\x00\\x00'\nt(g1\n(cpandas.core.index\nInt64Index\np6\n(I0\ntS'b'\ntRp7\n((I1\n(I3\ntg5\nI00\nS'\\x01\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x02\\x00\\x00\\x00\\x00\\x00\\x00\\x00\\x03\\x00\\x00\\x00\\x00\\x00\\x00\\x00'\nt(NttbNttb."
```

Is there a way to cast back to pandas.TimeSeries on the ipcluster engines?

Many thanks.



Discussion:
Pinging @wesm on this one.  There are really two issues here:
1. updating serialization to be aware of TimeSeries (and, I presume, data frames as a whole).
2. the interim workaround for how to reconstruct a TimeSeries from numpy arrays, if possible.

So the direct question to @wesm: Can you detail serialization of pandas objects?  What, in addition to the arrays themselves, is necessary to reconstruct a pandas object?  What pandas objects actually subclass ndarray (it is an `isinstance(cls, ndarray)` that is causing IPython to use its efficient numpy path here, rather than simply pickling).

I don't think this is a problem with pandas, I think it is more general than that because any class that subclasses numpy.ndarray will break under the current implementation. Therefore, I think it makes sense for the IPython.parallel code to only use the efficient numpy path if the object is

```
type(self.obj) == numpy.ndarray
```

rather than

```
isinstance(self.obj, numpy.ndarray)
```

Making DirectView.push efficient for pandas.TimeSeries objects is a nice-to-have, but I would have thought that it is ultimately less important than having DirectView.push work as expected.

As an interim workaround solution the pandas.TimeSeries object can be wrapped in a pandas.DataFrame to stop the efficient numpy path from being used.

Many thanks


Resolving patch:
diff --git a/IPython/utils/pickleutil.py b/IPython/utils/pickleutil.py
--- a/IPython/utils/pickleutil.py
+++ b/IPython/utils/pickleutil.py
@@ -208,6 +208,19 @@ def _import_mapping(mapping, original=None):
             else:
                 mapping[cls] = mapping.pop(key)
 
+def istype(obj, check):
+    """like isinstance(obj, check), but strict
+    
+    This won't catch subclasses.
+    """
+    if isinstance(check, tuple):
+        for cls in check:
+            if type(obj) is cls:
+                return True
+        return False
+    else:
+        return type(obj) is check
+
 def can(obj):
     """prepare an object for pickling"""
     
@@ -217,7 +230,7 @@ def can(obj):
         if isinstance(cls, basestring):
             import_needed = True
             break
-        elif isinstance(obj, cls):
+        elif istype(obj, cls):
             return canner(obj)
     
     if import_needed:


Test patch:
diff --git a/IPython/parallel/tests/test_view.py b/IPython/parallel/tests/test_view.py
--- a/IPython/parallel/tests/test_view.py
+++ b/IPython/parallel/tests/test_view.py
@@ -308,6 +308,21 @@ def test_push_pull_recarray(self):
         self.assertEqual(R2.dtype, R.dtype)
         self.assertEqual(R2.shape, R.shape)
         assert_array_equal(R2, R)
+
+    @skip_without('pandas')
+    def test_push_pull_timeseries(self):
+        """push/pull pandas.TimeSeries"""
+        import pandas
+        
+        ts = pandas.TimeSeries(range(10))
+        
+        view = self.client[-1]
+        
+        view.push(dict(ts=ts), block=True)
+        rts = view['ts']
+        
+        self.assertEqual(type(rts), type(ts))
+        self.assertTrue((ts == rts).all())
     
     def test_map(self):
         view = self.client[:]