Repository: bokeh/bokeh

Issue:
Bokeh Datatable renders dates one day off.
I am using Bokeh 0.11.0

When I produce a data table with dates, the rendered dates are one day off from the source of the data.  Taking the example from the bokeh documentation page (http://bokeh.pydata.org/en/0.11.1/docs/user_guide/interaction.html). 

![capture](https://cloud.githubusercontent.com/assets/18351854/14388887/d0363b40-fd75-11e5-9c6b-07bf3120db90.PNG)

If you look at the actual dates inside data, it should be 04/01/2014 to 03/102/014 (not 02/28/2014 to 03/09/2014).

![capture2](https://cloud.githubusercontent.com/assets/18351854/14388973/2682b26c-fd76-11e5-99b3-45ba58ee7240.PNG)



Discussion:
I wonder if this is actually some kind of timezone formatting issue. 

Work around is to just format date into a string with strftime and not use DateFormatter().

@bokeh/dev another thought is that a pure date starts exactly at midnight, and some kind of rounding error during serialization is causing the "milliseconds since epoch" to end up just slightly before midnight (and so on the previous day)

@sophiajiyoung can you try setting the `.format` field of the table cell formatter to `"@"` and report what is shown in the bokeh table? that will show the "ms since epoch" and we can start a comparison from there. 

For reference:

```
In [8]: d = date(2014, 3, 1)

In [9]: int(d.strftime("%s")) * 1000
Out[9]: 1393653600000
```

If printing the `ms` in the bokeh table shows something less than that that would explain things. 

I didn't realize the code was in the user's guide, I went ahead and made the change, there's definitely a discrepancy:
<img width="452" alt="screen shot 2016-04-21 at 5 03 39 pm" src="https://cloud.githubusercontent.com/assets/1078448/14725667/05d3e168-07e3-11e6-9bd5-632c94c1e135.png">

So:

```
In [13]: (1393653600000 - 1393632000000 ) / 1000 / 60 / 60
Out[13]: 6.0
```

It looks like what JS is reporting is exactly 6 hours too early?

OK here is the issue, we are converting to GMT:

https://github.com/bokeh/bokeh/blob/master/bokeh/core/json_encoder.py#L43-L47

@bokeh/dev anyone know the rationale here I am not sure that is the appropriate thing to do. 

I _think_ this was the commit by @kdodia: https://github.com/bokeh/bokeh/commit/6a8394460ecba3ff7a8f2484cc3b8639abdf0279

I think this is absolutely a bug, and that we should do this instead:

```
In [9]: time.mktime(d.timetuple()) * 1000
Out[9]: 1393653600000.0
```

ping @havocp @mattpap @bokeh/dev thoughts extremely appreciated on this one.

If the remainder of our code is not TZ aware (which I'm pretty sure it isn't), ~~then this should be definitively changed~~.

@mattpap to be clear you mean "the value should not be changed to GMT", not "the code should not be changed", correct?

Mmm, this has been an unresolved question for a while: https://github.com/bokeh/bokeh/issues/1135#issuecomment-54538225

> The issue here is that serialization of datetimes from Bokeh to BokehJS is not well-defined. (We have a couple related issues at #729 and #1103 if you want to see the conversations so far.)
> 
> At the moment BokehJS is timezone-agnostic and defaults to milliseconds from UTC epoch as its concept of a datetime. In your case, you have timezone-aware datetimes which are displayed with an offset because BokehJS is missing the fact that these are not, in fact, in UTC.

Now I'm confused, because if we leave non-UTC times then how bokehjs will now how to interpret a value?

https://github.com/bokeh/bokeh/issues/1135 may be related or a dupe, I am not quite sure. 

> may be related or a dupe

Definitively related, almost a duplicate.

Maybe the issue is that `timegm` does not do what is expected? it seems like it does not convert _to_ GMT, assumes the time _is_ GMT. That's why the returned time is 6 hrs early for me, for instance.

Edit:

```
In [12]: calendar.timegm?
Signature: calendar.timegm(tuple)
Docstring: Unrelated but handy function to calculate Unix timestamp from GMT.
File:      ~/anaconda/lib/python3.4/calendar.py
Type:      function
```

For some reason, `timetuple()` returns time in UTC (at least for me). Note there is also `utctimetuple()`. Anyway, at such a late time I shouldn't mess with time zones.

I guess there is an open question of should happen when the input _does_ have timezone information. But this is not about that (or at least part of this issue is not). It's about what to do when the data _does not_ have timezone specified, e.g. a  bare `datetime.date`. Can we agree on this statement:

> If there is no datetime specified the input data should be sent exactly as-is

And if that is that case, then the `timegm` call must be removed? 

@mattpap really? I get:

```
In [15]: d = date(2014, 3, 1)

In [16]: d.timetuple()
Out[16]: time.struct_time(tm_year=2014, tm_mon=3, tm_mday=1, tm_hour=0, tm_min=0, tm_sec=0, tm_wday=5, tm_yday=60, tm_isdst=-1)
```

Or as an alternative, make it configurable with an `assume_gmt` flag or something...


Resolving patch:
diff --git a/bokeh/core/json_encoder.py b/bokeh/core/json_encoder.py
--- a/bokeh/core/json_encoder.py
+++ b/bokeh/core/json_encoder.py
@@ -6,10 +6,10 @@
 import logging
 log = logging.getLogger(__name__)
 
-import calendar
 import datetime as dt
 import decimal
 import json
+import time
 
 import numpy as np
 
@@ -41,10 +41,10 @@ def transform_python_types(self, obj):
         # Datetime
         # datetime is a subclass of date.
         elif isinstance(obj, dt.datetime):
-            return calendar.timegm(obj.timetuple()) * 1000. + obj.microsecond / 1000.
+            return time.mktime(obj.timetuple()) * 1000. + obj.microsecond / 1000.
         # Date
         elif isinstance(obj, dt.date):
-            return calendar.timegm(obj.timetuple()) * 1000.
+            return time.mktime(obj.timetuple()) * 1000.
         # Numpy datetime64
         elif isinstance(obj, np.datetime64):
             epoch_delta = obj - np.datetime64('1970-01-01T00:00:00Z')


Test patch:
diff --git a/bokeh/core/tests/test_json_encoder.py b/bokeh/core/tests/test_json_encoder.py
--- a/bokeh/core/tests/test_json_encoder.py
+++ b/bokeh/core/tests/test_json_encoder.py
@@ -3,6 +3,9 @@
 import unittest
 from unittest import skipIf
 
+import datetime as dt
+import time
+
 import numpy as np
 
 try:
@@ -91,7 +94,7 @@ def test_nans_and_infs_pandas(self):
         assert deserialized[3] == 0
 
     @skipIf(not is_pandas, "pandas does not work in PyPy.")
-    def test_datetime_types(self):
+    def test_pandas_datetime_types(self):
         """should convert to millis
         """
         idx = pd.date_range('2001-1-1', '2001-1-5')
@@ -112,5 +115,19 @@ def test_datetime_types(self):
         }
         assert deserialized == baseline
 
+    def test_builtin_datetime_types(self):
+        """ should convert to millis as-is
+        """
+
+        a = dt.date(2016, 4, 28)
+        b = dt.datetime(2016, 4, 28, 2, 20, 50)
+        serialized = self.serialize({'a' : [a],
+                                     'b' : [b]})
+        deserialized = self.deserialize(serialized)
+
+        baseline = {u'a': [time.mktime(a.timetuple())*1000],
+                    u'b': [time.mktime(b.timetuple())*1000],
+        }
+        assert deserialized == baseline
 if __name__ == "__main__":
     unittest.main()