Repository: pandas-dev/pandas
Pull request: Test for #16726. unittest that ensures datetime is understood
 - [x] closes #16726
 - [x] test added


Problem statement:
Different behaviour on two different environments. TypeError: data type "datetime" not understood
#### I have an aggregation logic

```python
In [19]: datewise_prices
Out[19]: 
[{'arrivalDate': 1490227200000,
  'maxPrice': 3300,
  'minPrice': 3300,
  'modalPrice': 3300},
 {'arrivalDate': 1490400000000,
  'maxPrice': 3300,
  'minPrice': 3300,
  'modalPrice': 3300},
 {'arrivalDate': 1490832000000,
  'maxPrice': 3500,
  'minPrice': 3500,
  'modalPrice': 3500},
 {'arrivalDate': 1490918400000,
  'maxPrice': 3300,
  'minPrice': 3300,
  'modalPrice': 3300},
 {'arrivalDate': 1491091200000,
  'maxPrice': 3300,
  'minPrice': 3300,
  'modalPrice': 3300}]

In [20]: weekly_dataframe = pandas.DataFrame(datewise_prices)
    ...: weekly_dataframe.drop('minPrice', axis=1, inplace=True)
    ...: weekly_dataframe.drop('maxPrice', axis=1, inplace=True)
    ...: weekly_dataframe['arrivalDate'] = pandas.to_datetime(weekly_dataframe['arrivalDate'], unit='ms')
    ...: weekly_dataframe = weekly_dataframe.resample('W', on='arrivalDate')['modalPrice'].mean().dropna().reset_index()
    ...: weekly_dataframe['label'] = (weekly_dataframe['arrivalDate'] - pandas.offsets.DateOffset(days=6)).dt.strftime('%d %b') + ' to ' + weekly_dataframe['arriv
    ...: alDate'].dt.strftime('%d %b')

```
#### It results in an error on my Staging environment but works fine on my local.

The staging environment gives an error in `(weekly_dataframe['arrivalDate'] - pandas.offsets.DateOffset(days=6)).dt.strftime('%d %b')`

```
Traceback (most recent call last):
  File "/root/myapp/myapp/handlers/aggregation_handler.py", line 290, in get_daily_weekly_and_monthly_aggregates
    weekly_dataframe['label'] = (weekly_dataframe['arrivalDate'] - pandas.offsets.DateOffset(days=6)).dt.strftime('%d %b') + ' to ' + weekly_dataframe['arrivalDate'].dt.strftime('%d %b')
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/ops.py", line 696, in wrapper
    converted = _Op.get_op(left, right, name, na_op)
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/ops.py", line 330, in get_op
    return _TimeOp(left, right, name, na_op)
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/ops.py", line 343, in __init__
    lvalues = self._convert_to_array(left, name=name)
  File "/usr/local/lib/python2.7/dist-packages/pandas/core/ops.py", line 454, in _convert_to_array
    if (inferred_type in ('datetime64', 'datetime', 'date', 'time') or
TypeError: data type "datetime" not understood
```

but it works fine on my local.

#### Output of ``pd.show_versions()`` on local

<details>
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 4.4.0-78-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_IN
LOCALE: None.None

pandas: 0.19.2
nose: None
pip: 9.0.1
setuptools: 36.0.1
Cython: None
numpy: 1.12.1
scipy: None
statsmodels: None
xarray: None
IPython: 5.4.1
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2016.7
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: 0.9.6
lxml: None
bs4: None
html5lib: 0.999
httplib2: 0.8
apiclient: None
sqlalchemy: None
pymysql: None
psycopg2: None
jinja2: 2.8
boto: 2.40.0
pandas_datareader: None
</details>

#### Output of ``pd.show_versions()`` on Staging
<details>
INSTALLED VERSIONS
------------------
commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Linux
OS-release: 3.13.0-57-generic
machine: x86_64
processor: x86_64
byteorder: little
LC_ALL: None
LANG: en_US.UTF-8
LOCALE: None.None

pandas: 0.19.2
nose: None
pip: 1.5.4
setuptools: 3.3
Cython: None
numpy: 1.13.0
scipy: None
statsmodels: None
xarray: None
IPython: 4.0.0
sphinx: None
patsy: None
dateutil: 2.5.3
pytz: 2017.2
blosc: None
bottleneck: None
tables: None
numexpr: None
matplotlib: None
openpyxl: None
xlrd: None
xlwt: None
xlsxwriter: 0.9.6
lxml: None
bs4: None
html5lib: 0.999
httplib2: None
apiclient: None
sqlalchemy: 1.0.8
pymysql: None
psycopg2: None
jinja2: 2.9.6
boto: 2.40.0
pandas_datareader: None
</details>

Please help me out.

Hints from issue discussion:
Can you make a copy-pastable example, as requested?
```
import pandas as pd

datewise_prices = [
    {'arrivalDate': 1490227200000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300},
    {'arrivalDate': 1490400000000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300},
    {'arrivalDate': 1490832000000,
    'maxPrice': 3500,
    'minPrice': 3500,
    'modalPrice': 3500},
    {'arrivalDate': 1490918400000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300},
    {'arrivalDate': 1491091200000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300}
]


weekly_dataframe = pd.DataFrame(datewise_prices)
weekly_dataframe.drop('minPrice', axis=1, inplace=True)
weekly_dataframe.drop('maxPrice', axis=1, inplace=True)
weekly_dataframe['arrivalDate'] = pd.to_datetime(weekly_dataframe['arrivalDate'], unit='ms')
weekly_dataframe = weekly_dataframe.resample('W', on='arrivalDate')['modalPrice'].mean().dropna().reset_index()
weekly_dataframe['label'] = (weekly_dataframe['arrivalDate'] - pd.offsets.DateOffset(days=6)).dt.strftime('%d %b') + ' to ' + weekly_dataframe['arrivalDate'].dt.strftime('%d %b')
```

Transferred from @hussaintamboli 's input. It will help if you provide more information, for example, the full trace when it error.
@BranYang and @TomAugspurger, I have added the stacktrace.
Just figured it out myself.

It's because of numpy==1.13.0. For numpy==1.12.1, it's working fine.
this looks fine. what exactly is the issue?
On staging environment with the dependencies mentioned in `Output of pd.show_versions() on Staging`, I get `TypeError: data type "datetime" not understood`. 

But on local environment with dependencies mentioned in `Output of pd.show_versions() on local`, I get desired output.
your example does not repro. I would guess that you have strings (and not datetimes) in one of your fields
Reproduction steps.

This works

```
$ pip freeze | grep numpy 
numpy==1.12.1

import pandas as pd

datewise_prices = [
    {'arrivalDate': 1490227200000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300},
    {'arrivalDate': 1490400000000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300},
    {'arrivalDate': 1490832000000,
    'maxPrice': 3500,
    'minPrice': 3500,
    'modalPrice': 3500},
    {'arrivalDate': 1490918400000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300},
    {'arrivalDate': 1491091200000,
    'maxPrice': 3300,
    'minPrice': 3300,
    'modalPrice': 3300}
]


weekly_dataframe = pd.DataFrame(datewise_prices)
weekly_dataframe.drop('minPrice', axis=1, inplace=True)
weekly_dataframe.drop('maxPrice', axis=1, inplace=True)
weekly_dataframe['arrivalDate'] = pd.to_datetime(weekly_dataframe['arrivalDate'], unit='ms')
weekly_dataframe = weekly_dataframe.resample('W', on='arrivalDate')['modalPrice'].mean().dropna().reset_index()
weekly_dataframe['label'] = (weekly_dataframe['arrivalDate'] - pd.offsets.DateOffset(days=6)).dt.strftime('%d %b') + ' to ' + weekly_dataframe['arrivalDate'].dt.strftime('%d %b')

weekly_dataframe.drop('arrivalDate', axis=1, inplace=True)

print weekly_dataframe.to_dict(orient='records')
[{'label': u'20 Mar to 26 Mar', 'modalPrice': 3300.0},
 {'label': u'27 Mar to 02 Apr', 'modalPrice': 3366.6666666666665}]


```

On the staging environment where it gives error

```
pip install numpy==1.13.0

weekly_dataframe = pd.DataFrame(datewise_prices)
weekly_dataframe.drop('minPrice', axis=1, inplace=True)
weekly_dataframe.drop('maxPrice', axis=1, inplace=True)
weekly_dataframe['arrivalDate'] = pd.to_datetime(weekly_dataframe['arrivalDate'], unit='ms')
weekly_dataframe = weekly_dataframe.resample('W', on='arrivalDate')['modalPrice'].mean().dropna().reset_index()
weekly_dataframe['label'] = (weekly_dataframe['arrivalDate'] - pd.offsets.DateOffset(days=6)).dt.strftime('%d %b') + ' to ' + weekly_dataframe['arrivalDate'].dt.strftime('%d %b')

TypeError: data type "datetime" not understood

```
this works on master for both numpies so not sure what your issue is
I can reproduce it on 0.19.2 / 1.13.0. Small reproducible example:

```
pd.Series(pd.date_range("2012-01-01", periods=3)) - pd.offsets.DateOffset(days=6)
```

and the error actually comes from

```
np.dtype('M8[ns]') in ('datetime', 'datetime64')
```

But on master / 0.20.2, this does not raise anymore, so I suppose we have already fixed this (this line of code is also removed). 
Nonetheless, always welcome to do a PR to add a test for this specific case (the small reproducible example I showed above), to make sure it keeps on working.

Commit message: Test for #16726. unittest that ensures datetime is understood

Patch to review:
diff --git a/pandas/tests/series/test_datetime_values.py b/pandas/tests/series/test_datetime_values.py
index e1fc9af0cca89..456d337ce2b1c 100644
--- a/pandas/tests/series/test_datetime_values.py
+++ b/pandas/tests/series/test_datetime_values.py
@@ -409,3 +409,11 @@ def test_date_tz(self):
                            date(2015, 11, 22)])
         assert_series_equal(s.dt.date, expected)
         assert_series_equal(s.apply(lambda x: x.date()), expected)
+
+    def test_datetime_understood(self):
+        # Ensures it doesn't throw an exception reported in #16726
+        try:
+            pd.Series(pd.date_range("2012-01-01", periods=3)) - pd.offsets.DateOffset(days=6)
+        except Exception:
+            assert 'data type "datetime" not understood'
+


Code review comments:
diff_hunk: @@ -409,3 +409,11 @@ def test_date_tz(self):
                            date(2015, 11, 22)])
         assert_series_equal(s.dt.date, expected)
         assert_series_equal(s.apply(lambda x: x.date()), expected)
+
+    def test_datetime_understood(self):
+        # Ensures it doesn't throw an exception reported in #16726
+        try:
+            pd.Series(pd.date_range("2012-01-01", periods=3)) - pd.offsets.DateOffset(days=6)
+        except Exception:
+            assert 'data type "datetime" not understood'
path: pandas/tests/series/test_datetime_values.py
text: @user1:
This test will always pass, since

- if there's no exception originally, everything is fine
- If there is an exception, the `assert 'data type "datetime" not understood'` is True, so no AssertionError is raised

It's best to assign `result = ` that operation, construct an expected series `expected = pd.Series(pd.to_datetime(['2011-12-26', '2011-12-27', '2011-12-28'])` and then compare them with `tm.assert_series_equal(result, expected)`

@author:
You are right. My bad!

Please check now.