Repository: pandas-dev/pandas

Issue:
to_excel incorrectly sets time to midnight for exported datetime column
Hello,

I try this:

```
import requests
from pandas.io.json import json_normalize
import json
import pandas as pd
response = requests.get("http://api.openweathermap.org/data/2.5/history/station?start=1356220800&end=1356307200&type=hour&id=5530")
df = json_normalize(json.loads(response.text)['list'])
df['dt'] = pd.to_datetime(df['dt'], unit='s')
#df = df.set_index('dt')
print(df)
df.to_excel("out.xls")
```

It displays:

```
                    dt  humidity.c      ...       pressure.v  temp.c  temp.ma  \
0  2012-12-23 00:00:00           2      ...           1017.0       2   286.15
1  2012-12-23 01:00:00           2      ...           1017.0       2   286.15
2  2012-12-23 02:00:00           2      ...           1017.0       2   285.15
3  2012-12-23 03:00:00           2      ...           1017.0       2   284.15
4  2012-12-23 04:00:00           2      ...           1017.0       2   283.15
...
19 2012-12-23 19:00:00           1      ...           1016.0       1   282.15
20 2012-12-23 20:00:00           1      ...           1015.0       1   281.15
21 2012-12-23 21:00:00           1      ...           1015.0       1   282.15
22 2012-12-23 22:00:00           1      ...           1015.0       1   281.15
23 2012-12-23 23:00:00           1      ...           1005.0       1   286.15
24 2012-12-24 00:00:00           2      ...           1013.5       2   280.15
```

but if we look at Excel file `dt` column looks like

```
dt
2012-12-23 00:00:00
2012-12-23 00:00:00
2012-12-23 00:00:00
2012-12-23 00:00:00
2012-12-23 00:00:00
...
2012-12-23 00:00:00
2012-12-23 00:00:00
2012-12-23 00:00:00
2012-12-23 00:00:00
2012-12-24 00:00:00
```

`to_csv` method is ok

```
In [1]: pd.__version__
Out[1]: '0.15.2'
```

Kind regards



Discussion:
@scls19fr Thanks for the report!

Pandas supports a couple of external modules for writing Excel modules, so this might also depend on which one you're using.

Here is where you can find the relevant source code if you're interested in trying your hand at a fix: https://github.com/pydata/pandas/blob/master/pandas/io/excel.py

This was also reported here: http://stackoverflow.com/questions/27548833/pandas-to-excel-in-version-0-15-2-not-working-for-datetime-objects

But, as I posted over there, I cannot reproduce this. @scls19fr Can you show `pd.show_versions()`? @shoyer can you reproduce this?

```
pd.show_versions()

INSTALLED VERSIONS
------------------
commit: None
python: 2.7.6.final.0
python-bits: 64
OS: Darwin
OS-release: 14.0.0
machine: x86_64
processor: i386
byteorder: little
LC_ALL: None
LANG: fr_FR.UTF-8

pandas: 0.15.2
nose: 1.3.4
Cython: 0.21.1
numpy: 1.9.1
scipy: 0.14.1rc1
statsmodels: 0.6.1
IPython: 2.3.1
sphinx: 1.2.3
patsy: 0.3.0
dateutil: 2.3
pytz: 2014.9
bottleneck: None
tables: 3.1.1
numexpr: 2.4
matplotlib: 1.4.2
openpyxl: 2.0.3
xlrd: 0.9.3
xlwt: 0.7.5
xlsxwriter: 0.6.4
lxml: 3.4.1
bs4: 4.3.2
html5lib: 0.999
httplib2: 0.8
apiclient: None
rpy2: None
sqlalchemy: 0.9.8
pymysql: 0.6.2.None
psycopg2: None
```

I did

```
import requests
from pandas.io.json import json_normalize
import json
import pandas as pd
response = requests.get("http://api.openweathermap.org/data/2.5/history/station?start=1356220800&end=1356307200&type=hour&id=5530")
df = json_normalize(json.loads(response.text)['list'])
df['dt'] = pd.to_datetime(df['dt'], unit='s')
#df = df.set_index('dt')
df = df[['dt', 'main.temp.ma']]
print(df)


for (ext, engine) in [('xlsx', 'openpyxl'), ('xlsx', 'xlsxwriter'), ('xls', 'xlwt')]:
    with pd.ExcelWriter("out_{}.{}".format(engine, ext), engine=engine) as writer:
        df.to_excel(writer)
```

`out_openpyxl.xlsx` is ok but `out_xlsxwriter.xlsx` and `out_xlwt.xls` are bad (midnight issue).

@scls19fr Thanks, I can now confirm the same: working for `openpyxl`, wrong for `xlsxwriter` and `xlwt`

cc @jtratner @neirbowj

I am the author of XlsxWriter. Here is my analysis of what is happening in this issue starting with the smaller example above:

``` python
import requests
from pandas.io.json import json_normalize
import json
import pandas as pd
response = requests.get("http://api.openweathermap.org/data/2.5/history/station?start=1356220800&end=1356307200&type=hour&id=5530")

df = json_normalize(json.loads(response.text)['list'])
df['dt'] = pd.to_datetime(df['dt'], unit='s')
df = df[['dt', 'main.temp.ma']]

```

If we take one of these values as an example:

``` python
>>> timestamp = df['dt'][23]
>>> timestamp
Timestamp('2012-12-23 23:00:00')
```

An Excel date is stored as a delta of days from an epoch (the integer part of the number) and a percentage of the day for the time in milliseconds (the float part). XlsxWriter and afaik Xlwt use a calculation like the following:

``` python
>>> import datetime
>>> epoch = datetime.datetime(1899, 12, 31, 0, 0, 0)
>>> excel_datetime = timestamp - epoch
>>> excel_datetime
Timedelta('41265 days 23:00:00')
>>> excel_datetime.days()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'int' object is not callable
>>> excel_datetime.days
41265
>>> excel_datetime.seconds
0
>>> excel_datetime.microseconds
0
```

As you can see the seconds part of the `Timedelta()` doesn't look right and the seconds and microseconds are 0. Hence the 0 times in the Excel files.

For comparison here is how a `datetime.datatime` object behaves:

``` python
>>> mydate = datetime.datetime(2012, 12, 23, 23, 0, 0)
>>> excel_datetime = mydate - epoch
>>> excel_datetime
datetime.timedelta(41265, 82800)
>>> excel_datetime.days
41265
>>> excel_datetime.seconds
82800
>>> excel_datetime.microseconds
0
```

XlsxWriter performs this epoch calculation because the `Timestamp` object meets the following criteria:

``` python
>>> isinstance(timestamp, datetime.datetime)
True
```

So for me it looks like the issue here is that the pandas `Timestamp()` object doesn't handle the epoch calculation correctly.

There are a few workable solutions but I'll hold off until someone familiar with the `Timestamp` class has a chance to look at this and respond.

John

So the problem is the conversion is relying upon an implementation detail of `timedelta`, meaning that `.days` gets you the days and `.seconds` the seconds as the remainder. This was asof 0.15.0.
I noted this specifically here: http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#new-features.

There are many ways to do this. 

```
In [26]: ts = Timestamp('2012-12-23 23:00:00')

In [27]: epoch = datetime.datetime(1899, 12, 31, 0, 0, 0)

In [28]: td = ts - epoch

In [29]: td
Out[29]: Timedelta('41265 days 23:00:00')

In [32]: (td-Timedelta(days,unit='d')).total_seconds()
Out[32]: 82800.0

In [33]: td.components
Out[33]: Components(days=41265, hours=23, minutes=0, seconds=0, milliseconds=0, microseconds=0, nanoseconds=0)


In [35]: td.hours*3600+td.minutes*60+td.seconds
Out[35]: 82800

In [36]: td.to_pytimedelta().days
Out[36]: 41265

In [37]: td.to_pytimedelta().seconds
Out[37]: 82800
```

@jmcnamara This is very helpful, thanks!

For an explanation of how `pandas.Timedelta` works see this section of the docs (especially the warning):
http://pandas.pydata.org/pandas-docs/stable/timedeltas.html#attributes

This is certainly an unfortunate side-effect of overriding super-class behavior in a subclass. I recall raising this issue in the pull request adding `Timedelta`: https://github.com/pydata/pandas/pull/8184. It looks like @jreback convinced me that the overriding the super-class behavior was not so bad, but now I'm not so sure (since it clearly is relied in in external packages).

The simplest fix would be to use `to_pydatetime()` for coercing `datetime64` columns (prior to passing off to the various Excel writing backends)

FWIW, I think pandas should fix this issue on our side (since we broke it with our strange `timedelta` subclass) rather than asking xlsxwriter and xlwt to fix this upstream.

@shoyer @jreback Open a new issue to discuss this behaviour of the timedelta attributes again?

this is not an upstream issue
rather a bug in the pandss openpyxl engine 

@jreback wait... wat? 

I've added a PR for this based on coercing the `Timestamp` object to a `datetime` object for xlwt and xlsxwriter with a test.

Even if you choose a different solution you can use the testcase.

I was surprised that the existing testcases didn't pick up this issue but none of them actually test a roundtrip of a date and time. `o_O` But I am not in a position to throw the first stone on that one.

@neirbowj sorry, I meant xlwt/xlsxwriter


Resolving patch:
diff --git a/pandas/tseries/tdi.py b/pandas/tseries/tdi.py
--- a/pandas/tseries/tdi.py
+++ b/pandas/tseries/tdi.py
@@ -118,8 +118,8 @@ def _join_i8_wrapper(joinf, **kwargs):
     _left_indexer_unique = _join_i8_wrapper(
         _algos.left_join_indexer_unique_int64, with_indexers=False)
     _arrmap = None
-    _datetimelike_ops = ['days','hours','minutes','seconds','milliseconds','microseconds',
-                         'nanoseconds','freq','components']
+    _datetimelike_ops = ['days','seconds','microseconds','nanoseconds',
+                         'freq','components']
 
     __eq__ = _td_index_cmp('__eq__')
     __ne__ = _td_index_cmp('__ne__', nat_result=True)
@@ -349,37 +349,22 @@ def _get_field(self, m):
 
     @property
     def days(self):
-        """ The number of integer days for each element """
+        """ Number of days for each element. """
         return self._get_field('days')
 
-    @property
-    def hours(self):
-        """ The number of integer hours for each element """
-        return self._get_field('hours')
-
-    @property
-    def minutes(self):
-        """ The number of integer minutes for each element """
-        return self._get_field('minutes')
-
     @property
     def seconds(self):
-        """ The number of integer seconds for each element """
+        """ Number of seconds (>= 0 and less than 1 day) for each element. """
         return self._get_field('seconds')
 
-    @property
-    def milliseconds(self):
-        """ The number of integer milliseconds for each element """
-        return self._get_field('milliseconds')
-
     @property
     def microseconds(self):
-        """ The number of integer microseconds for each element """
+        """ Number of microseconds (>= 0 and less than 1 second) for each element. """
         return self._get_field('microseconds')
 
     @property
     def nanoseconds(self):
-        """ The number of integer nanoseconds for each element """
+        """ Number of nanoseconds (>= 0 and less than 1 microsecond) for each element. """
         return self._get_field('nanoseconds')
 
     @property


Test patch:
diff --git a/pandas/io/tests/test_excel.py b/pandas/io/tests/test_excel.py
--- a/pandas/io/tests/test_excel.py
+++ b/pandas/io/tests/test_excel.py
@@ -1151,6 +1151,29 @@ def test_swapped_columns(self):
             tm.assert_series_equal(write_frame['A'], read_frame['A'])
             tm.assert_series_equal(write_frame['B'], read_frame['B'])
 
+    def test_datetimes(self):
+
+        # Test writing and reading datetimes. For issue #9139. (xref #9185)
+        _skip_if_no_xlrd()
+
+        datetimes = [datetime(2013, 1, 13, 1, 2, 3),
+                     datetime(2013, 1, 13, 2, 45, 56),
+                     datetime(2013, 1, 13, 4, 29, 49),
+                     datetime(2013, 1, 13, 6, 13, 42),
+                     datetime(2013, 1, 13, 7, 57, 35),
+                     datetime(2013, 1, 13, 9, 41, 28),
+                     datetime(2013, 1, 13, 11, 25, 21),
+                     datetime(2013, 1, 13, 13, 9, 14),
+                     datetime(2013, 1, 13, 14, 53, 7),
+                     datetime(2013, 1, 13, 16, 37, 0),
+                     datetime(2013, 1, 13, 18, 20, 52)]
+
+        with ensure_clean(self.ext) as path:
+            write_frame = DataFrame.from_items([('A', datetimes)])
+            write_frame.to_excel(path, 'Sheet1')
+            read_frame = read_excel(path, 'Sheet1', header=0)
+
+            tm.assert_series_equal(write_frame['A'], read_frame['A'])
 
 def raise_wrapper(major_ver):
     def versioned_raise_wrapper(orig_method):
diff --git a/pandas/tests/test_series.py b/pandas/tests/test_series.py
--- a/pandas/tests/test_series.py
+++ b/pandas/tests/test_series.py
@@ -84,7 +84,7 @@ def test_dt_namespace_accessor(self):
         ok_for_dt = ok_for_base + ['date','time','microsecond','nanosecond', 'is_month_start', 'is_month_end', 'is_quarter_start',
                                    'is_quarter_end', 'is_year_start', 'is_year_end', 'tz']
         ok_for_dt_methods = ['to_period','to_pydatetime','tz_localize','tz_convert']
-        ok_for_td = ['days','hours','minutes','seconds','milliseconds','microseconds','nanoseconds']
+        ok_for_td = ['days','seconds','microseconds','nanoseconds']
         ok_for_td_methods = ['components','to_pytimedelta']
 
         def get_expected(s, name):
diff --git a/pandas/tseries/tests/test_timedeltas.py b/pandas/tseries/tests/test_timedeltas.py
--- a/pandas/tseries/tests/test_timedeltas.py
+++ b/pandas/tseries/tests/test_timedeltas.py
@@ -301,15 +301,18 @@ class Other:
         self.assertTrue(td.__floordiv__(td) is NotImplemented)
 
     def test_fields(self):
+
+        # compat to datetime.timedelta
         rng = to_timedelta('1 days, 10:11:12')
         self.assertEqual(rng.days,1)
-        self.assertEqual(rng.hours,10)
-        self.assertEqual(rng.minutes,11)
-        self.assertEqual(rng.seconds,12)
-        self.assertEqual(rng.milliseconds,0)
+        self.assertEqual(rng.seconds,10*3600+11*60+12)
         self.assertEqual(rng.microseconds,0)
         self.assertEqual(rng.nanoseconds,0)
 
+        self.assertRaises(AttributeError, lambda : rng.hours)
+        self.assertRaises(AttributeError, lambda : rng.minutes)
+        self.assertRaises(AttributeError, lambda : rng.milliseconds)
+
         td = Timedelta('-1 days, 10:11:12')
         self.assertEqual(abs(td),Timedelta('13:48:48'))
         self.assertTrue(str(td) == "-1 days +10:11:12")
@@ -317,14 +320,14 @@ def test_fields(self):
         self.assertEqual(-Timedelta('-1 days, 10:11:12').value,49728000000000)
         self.assertEqual(Timedelta('-1 days, 10:11:12').value,-49728000000000)
 
-        rng = to_timedelta('-1 days, 10:11:12')
+        rng = to_timedelta('-1 days, 10:11:12.100123456')
         self.assertEqual(rng.days,-1)
-        self.assertEqual(rng.hours,10)
-        self.assertEqual(rng.minutes,11)
-        self.assertEqual(rng.seconds,12)
-        self.assertEqual(rng.milliseconds,0)
-        self.assertEqual(rng.microseconds,0)
-        self.assertEqual(rng.nanoseconds,0)
+        self.assertEqual(rng.seconds,10*3600+11*60+12)
+        self.assertEqual(rng.microseconds,100*1000+123)
+        self.assertEqual(rng.nanoseconds,456)
+        self.assertRaises(AttributeError, lambda : rng.hours)
+        self.assertRaises(AttributeError, lambda : rng.minutes)
+        self.assertRaises(AttributeError, lambda : rng.milliseconds)
 
         # components
         tup = pd.to_timedelta(-1, 'us').components
@@ -830,22 +833,22 @@ def test_astype(self):
         self.assert_numpy_array_equal(result, rng.asi8)
 
     def test_fields(self):
-        rng = timedelta_range('1 days, 10:11:12', periods=2, freq='s')
+        rng = timedelta_range('1 days, 10:11:12.100123456', periods=2, freq='s')
         self.assert_numpy_array_equal(rng.days, np.array([1,1],dtype='int64'))
-        self.assert_numpy_array_equal(rng.hours, np.array([10,10],dtype='int64'))
-        self.assert_numpy_array_equal(rng.minutes, np.array([11,11],dtype='int64'))
-        self.assert_numpy_array_equal(rng.seconds, np.array([12,13],dtype='int64'))
-        self.assert_numpy_array_equal(rng.milliseconds, np.array([0,0],dtype='int64'))
-        self.assert_numpy_array_equal(rng.microseconds, np.array([0,0],dtype='int64'))
-        self.assert_numpy_array_equal(rng.nanoseconds, np.array([0,0],dtype='int64'))
+        self.assert_numpy_array_equal(rng.seconds, np.array([10*3600+11*60+12,10*3600+11*60+13],dtype='int64'))
+        self.assert_numpy_array_equal(rng.microseconds, np.array([100*1000+123,100*1000+123],dtype='int64'))
+        self.assert_numpy_array_equal(rng.nanoseconds, np.array([456,456],dtype='int64'))
+
+        self.assertRaises(AttributeError, lambda : rng.hours)
+        self.assertRaises(AttributeError, lambda : rng.minutes)
+        self.assertRaises(AttributeError, lambda : rng.milliseconds)
 
         # with nat
         s = Series(rng)
         s[1] = np.nan
 
         tm.assert_series_equal(s.dt.days,Series([1,np.nan],index=[0,1]))
-        tm.assert_series_equal(s.dt.hours,Series([10,np.nan],index=[0,1]))
-        tm.assert_series_equal(s.dt.milliseconds,Series([0,np.nan],index=[0,1]))
+        tm.assert_series_equal(s.dt.seconds,Series([10*3600+11*60+12,np.nan],index=[0,1]))
 
     def test_components(self):
         rng = timedelta_range('1 days, 10:11:12', periods=2, freq='s')