Repository: bokeh/bokeh

Issue:
bokeh.util.serialization.py  convert_datetime_type breaks when passed timezone aware datetime
After updating to bokeh 0.12.6 the convert_datetime_type became unable to handle datetime objecs that are timezone aware.  The subtraction of DT_EPOCH which is a naive datetime from a obj which might be aware or naive fails when obj is timezone aware.    The error indicates that subtraction of a datetime object and a naive object is not supported.

This occurs with python 2.7.

proposed fix to serialization.py attached.

$ pip freeze
appdirs==1.4.3
awsebcli==3.7.7
backports-abc==0.4
backports.ssl-match-hostname==3.5.0.1
bkcharts==0.2
blessed==1.9.5
bokeh==0.12.6
botocore==1.4.40
cement==2.8.2
certifi==2016.2.28
colorama==0.3.7
cycler==0.10.0
DateTime==4.1.1
Django==1.9.8
docker-py==1.7.2
dockerpty==0.4.1
docopt==0.6.2
docutils==0.12
futures==3.0.5
Jinja2==2.8
jmespath==0.9.0
MarkupSafe==0.23
msgpack-python==0.4.8
numpy==1.11.1
packaging==16.8
pandas==0.18.1
pathspec==0.3.4
pyasn1==0.1.9
pygobject==3.18.2
pyparsing==2.1.5
python-dateutil==2.5.3
pytz==2016.6.1
PyYAML==3.11
requests==2.9.1
rsa==3.4.2
s3transfer==0.0.1
semantic-version==2.5.0
singledispatch==3.4.0.3
six==1.10.0
tabulate==0.7.5
td-client==0.5.0
termcolor==1.1.0
texttable==0.8.4
tornado==4.4.1
tzlocal==1.2.2
urllib3==1.17
virtualenv==15.0.2
wcwidth==0.1.7
websocket-client==0.37.0
zope.interface==4.2.0

[serialization.py.txt](https://github.com/bokeh/bokeh/files/1078483/serialization.py.txt)
 



Discussion:
@asteriskSF The proposed fix would be much easier to interpret as a diff. Can you possibly provide a diff, or open a PR for discussion? Discussion is definitely needed, as I seem to notice that your file unconditionally imports from `django`. I can already state that this would not be acceptable as-is, as we would not want to make Django a dependency of Bokeh. 
I did some more testing, and the fix I proposed didn't have the same behavior for time-aware datetime as in bokeh 0.12.5.  With a bit more testing, I found a much simpler way to solve the problem that doesn't require any additional dependencies and also gives similar output to bokeh 0.12.5!

Here is the diff:
```
104c104
<         return (obj - DT_EPOCH).total_seconds() * 1000. + obj.microsecond / 1000.
---
>         return (obj.replace(tzinfo=None) - DT_EPOCH).total_seconds() * 1000. + obj.microsecond / 1000.
```

This change drops the tzinfo making obj always timezone naive.  I suspect it will have problems if objects with different timezones are passed in, but that probably also existed in bokeh 0.12.5.

I found if timezone awareness was maintained the plot would be displayed in utc time, which was not the behavior previously and for me at least undesirable.  Perhaps bokeh should be displaying with system timezone or perhaps using the timezone of the corresponding axis?
hopefully a more experienced dev can take the diff and make the PR.  I'm not sure what testing infrastructure is in place for bokeh or how to proceed with the change.
@asteriskSF Because handling timezones is so difficult what we strive to do is to pass the date time information to BokehJS exactly as-is, without any tz transformations. Then if users wanted to adjust, they could use a custom tick formatter, etc. This seems the most likely to be maintainable as "correct" while still providing capability to users who need it. I *think* your proposed solution is exactly in line with that .

My only other question is about `obj.replace(tzinfo=None)` Does that return a new object? Or is does it mutate in place?

@bokeh/dev any thoughts?

Regarding a PR, I am happy to make one but I'd certainly also be happy to help you make one as well so that you are credited for it in the GH contributors page. If you'd like to pursue that, there's no hurry, and you can find a reasonable overview of making a PR here:

https://github.com/CPAN-PRC/resources/wiki/My-first-Pull-Request

Just let me know either way. :)


datetime.replace returns a new object

I'm not concerned with credit, please proceed with the PR without me.

OK I have marked it for `0.12.7` milestone, thanks for the thorough report and patch

Resolving patch:
diff --git a/bokeh/util/serialization.py b/bokeh/util/serialization.py
--- a/bokeh/util/serialization.py
+++ b/bokeh/util/serialization.py
@@ -101,7 +101,8 @@ def convert_datetime_type(obj):
 
     # Datetime (datetime is a subclass of date)
     elif isinstance(obj, dt.datetime):
-        return (obj - DT_EPOCH).total_seconds() * 1000. + obj.microsecond / 1000.
+        diff = obj.replace(tzinfo=None) - DT_EPOCH
+        return diff.total_seconds() * 1000. + obj.microsecond / 1000.
 
     # Timedelta (timedelta is class in the datetime library)
     elif isinstance(obj, dt.timedelta):


Test patch:
diff --git a/bokeh/util/tests/test_serialization.py b/bokeh/util/tests/test_serialization.py
--- a/bokeh/util/tests/test_serialization.py
+++ b/bokeh/util/tests/test_serialization.py
@@ -6,6 +6,7 @@
 import pytest
 import numpy as np
 import pandas as pd
+import pytz
 
 import bokeh.util.serialization as bus
 
@@ -60,6 +61,12 @@ def test_convert_datetime_type():
     assert bus.convert_datetime_type(pd.Timedelta("3000ms")) == 3000.0
     assert bus.convert_datetime_type(bus._pd_timestamp(3000000)) == 3.0
 
+def test_convert_datetime_type_with_tz():
+    # This ensures datetimes are sent to BokehJS timezone-naive
+    # see https://github.com/bokeh/bokeh/issues/6480
+    for tz in pytz.all_timezones:
+        assert bus.convert_datetime_type(datetime.datetime(2016, 5, 11, tzinfo=datetime.tzinfo(tz))) == 1462924800000.0
+
 testing = [[float('nan'), 3], [float('-inf'), [float('inf')]]]
 expected = [['NaN', 3.0], ['-Infinity', ['Infinity']]]