Repository: pandas-dev/pandas

Issue:
BUG: df.apply handles np.timedelta64 as timestamp, should be timedelta
I think there may be a bug with the row-wise handling of `numpy.timedelta64` data types when using `DataFrame.apply`. As a check, the problem does not appear when using `DataFrame.applymap`. The problem may be related to #4532, but I'm unsure. I've included an example below.

This is only a minor problem for my use-case, which is cross-checking timestamps from a counter/timer card. I can easily work around the issue with `DataFrame.itertuples` etc.

Thank you for your time and for making such a useful package!
#### Example
##### Version

Import and check versions.

```
$ date
Thu Jul 17 16:28:38 CDT 2014
$ conda update pandas
Fetching package metadata: ..
# All requested packages already installed.
# packages in environment at /Users/harrold/anaconda:
#
pandas                    0.14.1               np18py27_0  
$ ipython
Python 2.7.8 |Anaconda 2.0.1 (x86_64)| (default, Jul  2 2014, 15:36:00) 
Type "copyright", "credits" or "license" for more information.

IPython 2.1.0 -- An enhanced Interactive Python.
Anaconda is brought to you by Continuum Analytics.
Please check out: http://continuum.io/thanks and https://binstar.org
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: from __future__ import print_function

In [2]: import numpy as np

In [3]: import pandas as pd

In [4]: pd.util.print_versions.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.8.final.0
python-bits: 64
OS: Darwin
OS-release: 11.4.2
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8

pandas: 0.14.1
nose: 1.3.3
Cython: 0.20.1
numpy: 1.8.1
scipy: 0.14.0
statsmodels: 0.5.0
IPython: 2.1.0
sphinx: 1.2.2
patsy: 0.2.1
scikits.timeseries: None
dateutil: 1.5
pytz: 2014.4
bottleneck: None
tables: 3.1.1
numexpr: 2.3.1
matplotlib: 1.3.1
openpyxl: 1.8.5
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: 0.5.5
lxml: 3.3.5
bs4: 4.3.1
html5lib: 0.999
httplib2: 0.8
apiclient: 1.2
rpy2: None
sqlalchemy: 0.9.4
pymysql: None
psycopg2: None
```
##### Create test data

Using subset of original raw data as example.

```
In [5]: datetime_start = np.datetime64(u'2014-05-31T01:23:19.9600345Z')

In [6]: timedeltas_elapsed = [30053400, 40053249, 50053098]
```

Compute datetimes from elapsed timedeltas, then create differential timedeltas from datetimes. All elements are either type `numpy.datetime64` or `numpy.timedelta64`.

```
In [7]: df = pd.DataFrame(dict(datetimes = timedeltas_elapsed))

In [8]: df = df.applymap(lambda elt: np.timedelta64(elt, 'us'))

In [9]: df = df.applymap(lambda elt: np.datetime64(datetime_start + elt))

In [10]: df['differential_timedeltas'] = df['datetimes'] - df['datetimes'].shift()

In [11]: print(df)
                      datetimes  differential_timedeltas
0 2014-05-31 01:23:50.013434500                      NaT
1 2014-05-31 01:24:00.013283500          00:00:09.999849
2 2014-05-31 01:24:10.013132500          00:00:09.999849
```
##### Expected behavior

With element-wise handling using `DataFrame.applymap`, all elements are correctly identified as datetimes (timestamps) or timedeltas.

```
In [12]: print(df.applymap(lambda elt: type(elt)))
                          datetimes     differential_timedeltas
0  <class 'pandas.tslib.Timestamp'>  <type 'numpy.timedelta64'>
1  <class 'pandas.tslib.Timestamp'>  <type 'numpy.timedelta64'>
2  <class 'pandas.tslib.Timestamp'>  <type 'numpy.timedelta64'>
```
##### Bug

With row-wise handling using `DataFrame.apply`, all elements are type `pandas.tslib.Timestamp`. I expected 'differential_timedeltas' to be type `numpy.timedelta64` or another type of timedelta, not a type of datetime (timestamp).

```
In [13]: # For 'datetimes':

In [14]: print(df.apply(lambda row: type(row['datetimes']), axis=1))
0    <class 'pandas.tslib.Timestamp'>
1    <class 'pandas.tslib.Timestamp'>
2    <class 'pandas.tslib.Timestamp'>
dtype: object

In [15]: # For 'differential_timedeltas':

In [16]: print(df.apply(lambda row: type(row['differential_timedeltas']), axis=1))
0      <class 'pandas.tslib.NaTType'>
1    <class 'pandas.tslib.Timestamp'>
2    <class 'pandas.tslib.Timestamp'>
dtype: object
```



Discussion:
actually this is a sympton of a more insidius issue.

try `df.values`. This is a 'common-dtype' for the frame, unfortunately its wrong, it should be `object`, and NOT `datetime64[ns]`. So some logic messed up here: https://github.com/pydata/pandas/blob/master/pandas/core/internals.py#L3673

I think it needs a tiny tweek (and test cases of course!).

interested in a pull-request?

Thanks for the fast response!

I'll give it a go, but as a newbie this may be over my head to do in a time-efficient manner. Guess I'll start reading through http://pandas.pydata.org/developers.html ...

I'll reply to this thread when I have some progress on the patch. Thanks again.

If you have questions along the way don't hesitate to ask, we don't bite


Resolving patch:
diff --git a/pandas/core/frame.py b/pandas/core/frame.py
--- a/pandas/core/frame.py
+++ b/pandas/core/frame.py
@@ -3539,6 +3539,7 @@ def _apply_standard(self, func, axis, ignore_failures=False, reduce=True):
             except Exception:
                 pass
 
+        dtype = object if self._is_mixed_type else None
         if axis == 0:
             series_gen = (self.icol(i) for i in range(len(self.columns)))
             res_index = self.columns
@@ -3547,7 +3548,7 @@ def _apply_standard(self, func, axis, ignore_failures=False, reduce=True):
             res_index = self.index
             res_columns = self.columns
             values = self.values
-            series_gen = (Series.from_array(arr, index=res_columns, name=name)
+            series_gen = (Series.from_array(arr, index=res_columns, name=name, dtype=dtype)
                           for i, (arr, name) in
                           enumerate(zip(values, res_index)))
         else:  # pragma : no cover
diff --git a/pandas/core/internals.py b/pandas/core/internals.py
--- a/pandas/core/internals.py
+++ b/pandas/core/internals.py
@@ -25,7 +25,7 @@
 from pandas.util.decorators import cache_readonly
 
 from pandas.tslib import Timestamp
-from pandas import compat
+from pandas import compat, _np_version_under1p7
 from pandas.compat import range, map, zip, u
 from pandas.tseries.timedeltas import _coerce_scalar_to_timedelta_type
 
@@ -1290,6 +1290,16 @@ def to_native_types(self, slicer=None, na_rep=None, **kwargs):
         return rvalues.tolist()
 
 
+    def get_values(self, dtype=None):
+        # return object dtypes as datetime.timedeltas
+        if dtype == object:
+            if _np_version_under1p7:
+                return self.values.astype('object')
+            return lib.map_infer(self.values.ravel(),
+                                 lambda x: timedelta(microseconds=x.item()/1000)
+                                 ).reshape(self.values.shape)
+        return self.values
+
 class BoolBlock(NumericBlock):
     __slots__ = ()
     is_bool = True
@@ -2595,7 +2605,7 @@ def as_matrix(self, items=None):
         else:
             mgr = self
 
-        if self._is_single_block:
+        if self._is_single_block or not self.is_mixed_type:
             return mgr.blocks[0].get_values()
         else:
             return mgr._interleave()
@@ -3647,9 +3657,11 @@ def _lcd_dtype(l):
     has_non_numeric = have_dt64 or have_td64 or have_cat
 
     if (have_object or
-        (have_bool and have_numeric) or
+        (have_bool and (have_numeric or have_dt64 or have_td64)) or
         (have_numeric and has_non_numeric) or
-        have_cat):
+        have_cat or
+        have_dt64 or
+        have_td64):
         return np.dtype(object)
     elif have_bool:
         return np.dtype(bool)
@@ -3670,10 +3682,6 @@ def _lcd_dtype(l):
             return np.dtype('int%s' % (lcd.itemsize * 8 * 2))
         return lcd
 
-    elif have_dt64 and not have_float and not have_complex:
-        return np.dtype('M8[ns]')
-    elif have_td64 and not have_float and not have_complex:
-        return np.dtype('m8[ns]')
     elif have_complex:
         return np.dtype('c16')
     else:
diff --git a/pandas/core/series.py b/pandas/core/series.py
--- a/pandas/core/series.py
+++ b/pandas/core/series.py
@@ -237,14 +237,14 @@ def __init__(self, data=None, index=None, dtype=None, name=None,
         self._set_axis(0, index, fastpath=True)
 
     @classmethod
-    def from_array(cls, arr, index=None, name=None, copy=False,
+    def from_array(cls, arr, index=None, name=None, dtype=None, copy=False,
                    fastpath=False):
         # return a sparse series here
         if isinstance(arr, ABCSparseArray):
             from pandas.sparse.series import SparseSeries
             cls = SparseSeries
 
-        return cls(arr, index=index, name=name, copy=copy, fastpath=fastpath)
+        return cls(arr, index=index, name=name, dtype=dtype, copy=copy, fastpath=fastpath)
 
     @property
     def _constructor(self):


Test patch:
diff --git a/pandas/tests/test_frame.py b/pandas/tests/test_frame.py
--- a/pandas/tests/test_frame.py
+++ b/pandas/tests/test_frame.py
@@ -9635,6 +9635,15 @@ def test_apply(self):
             [[1, 2, 3], [4, 5, 6], [7, 8, 9]], index=['a', 'a', 'c'])
         self.assertRaises(ValueError, df.apply, lambda x: x, 2)
 
+    def test_apply_mixed_datetimelike(self):
+        tm._skip_if_not_numpy17_friendly()
+
+        # mixed datetimelike
+        # GH 7778
+        df = DataFrame({ 'A' : date_range('20130101',periods=3), 'B' : pd.to_timedelta(np.arange(3),unit='s') })
+        result = df.apply(lambda x: x, axis=1)
+        assert_frame_equal(result, df)
+
     def test_apply_empty(self):
         # empty
         applied = self.empty.apply(np.sqrt)
diff --git a/pandas/tests/test_internals.py b/pandas/tests/test_internals.py
--- a/pandas/tests/test_internals.py
+++ b/pandas/tests/test_internals.py
@@ -3,7 +3,7 @@
 import nose
 import numpy as np
 
-from pandas import Index, MultiIndex, DataFrame, Series
+from pandas import Index, MultiIndex, DataFrame, Series, Categorical
 from pandas.compat import OrderedDict, lrange
 from pandas.sparse.array import SparseArray
 from pandas.core.internals import *
@@ -41,9 +41,11 @@ def create_block(typestr, placement, item_shape=None, num_offset=0):
         * complex, c16, c8
         * bool
         * object, string, O
-        * datetime, dt
+        * datetime, dt, M8[ns]
+        * timedelta, td, m8[ns]
         * sparse (SparseArray with fill_value=0.0)
         * sparse_na (SparseArray with fill_value=np.nan)
+        * category, category2
 
     """
     placement = BlockPlacement(placement)
@@ -67,8 +69,14 @@ def create_block(typestr, placement, item_shape=None, num_offset=0):
                             shape)
     elif typestr in ('bool'):
         values = np.ones(shape, dtype=np.bool_)
-    elif typestr in ('datetime', 'dt'):
+    elif typestr in ('datetime', 'dt', 'M8[ns]'):
         values = (mat * 1e9).astype('M8[ns]')
+    elif typestr in ('timedelta', 'td', 'm8[ns]'):
+        values = (mat * 1).astype('m8[ns]')
+    elif typestr in ('category'):
+        values = Categorical([1,1,2,2,3,3,3,3,4,4])
+    elif typestr in ('category2'):
+        values = Categorical(['a','a','a','a','b','b','c','c','c','d'])
     elif typestr in ('sparse', 'sparse_na'):
         # FIXME: doesn't support num_rows != 10
         assert shape[-1] == 10
@@ -556,7 +564,54 @@ def _compare(old_mgr, new_mgr):
         self.assertEqual(new_mgr.get('h').dtype, np.float16)
 
     def test_interleave(self):
-        pass
+
+
+        # self
+        for dtype in ['f8','i8','object','bool','complex','M8[ns]','m8[ns]']:
+            mgr = create_mgr('a: {0}'.format(dtype))
+            self.assertEqual(mgr.as_matrix().dtype,dtype)
+            mgr = create_mgr('a: {0}; b: {0}'.format(dtype))
+            self.assertEqual(mgr.as_matrix().dtype,dtype)
+
+        # will be converted according the actual dtype of the underlying
+        mgr = create_mgr('a: category')
+        self.assertEqual(mgr.as_matrix().dtype,'i8')
+        mgr = create_mgr('a: category; b: category')
+        self.assertEqual(mgr.as_matrix().dtype,'i8'),
+        mgr = create_mgr('a: category; b: category2')
+        self.assertEqual(mgr.as_matrix().dtype,'object')
+        mgr = create_mgr('a: category2')
+        self.assertEqual(mgr.as_matrix().dtype,'object')
+        mgr = create_mgr('a: category2; b: category2')
+        self.assertEqual(mgr.as_matrix().dtype,'object')
+
+        # combinations
+        mgr = create_mgr('a: f8')
+        self.assertEqual(mgr.as_matrix().dtype,'f8')
+        mgr = create_mgr('a: f8; b: i8')
+        self.assertEqual(mgr.as_matrix().dtype,'f8')
+        mgr = create_mgr('a: f4; b: i8')
+        self.assertEqual(mgr.as_matrix().dtype,'f4')
+        mgr = create_mgr('a: f4; b: i8; d: object')
+        self.assertEqual(mgr.as_matrix().dtype,'object')
+        mgr = create_mgr('a: bool; b: i8')
+        self.assertEqual(mgr.as_matrix().dtype,'object')
+        mgr = create_mgr('a: complex')
+        self.assertEqual(mgr.as_matrix().dtype,'complex')
+        mgr = create_mgr('a: f8; b: category')
+        self.assertEqual(mgr.as_matrix().dtype,'object')
+        mgr = create_mgr('a: M8[ns]; b: category')
+        self.assertEqual(mgr.as_matrix().dtype,'object')
+        mgr = create_mgr('a: M8[ns]; b: bool')
+        self.assertEqual(mgr.as_matrix().dtype,'object')
+        mgr = create_mgr('a: M8[ns]; b: i8')
+        self.assertEqual(mgr.as_matrix().dtype,'object')
+        mgr = create_mgr('a: m8[ns]; b: bool')
+        self.assertEqual(mgr.as_matrix().dtype,'object')
+        mgr = create_mgr('a: m8[ns]; b: i8')
+        self.assertEqual(mgr.as_matrix().dtype,'object')
+        mgr = create_mgr('a: M8[ns]; b: m8[ns]')
+        self.assertEqual(mgr.as_matrix().dtype,'object')
 
     def test_interleave_non_unique_cols(self):
         df = DataFrame([