Repository: bokeh/bokeh

Issue:
Creating a ColumnDataSource with a DataFrame makes it impossible to use streaming
Bokeh 0.12.6

Executing
```python
from bokeh.models import ColumnDataSource
import pandas as pd

d = pd.DataFrame({'a': [1]})
cds = ColumnDataSource(d)
cds.stream(ColumnDataSource.from_df(d))
```
leads to
```
AttributeError: 'Series' object has no attribute 'extend'
```

The reason is that `ColumnDataSource._data_from_df` uses `DataFrame.to_dict('series')` but `PropertyValueDict._stream` assumes the columns to be lists, or at least something that has `extend` method.

On a related note - probably making `steam` also accept a raw `DataFrame` would make the interface more cohesive.



Discussion:
@p-himik Thanks for the report. Would you be interested in working up a small PR to improve things? The current core team is pretty swamped so I can almost certainly guarantee that you would be able to get to this before we would. The only real complication is that we don't want to introduce a hard dependency on pandas, so the code will need to use our `optional_import` to see if pandas is present before type checking (or else maybe a duck typing check for [`Series.append`](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.Series.append.html) would work too) 
I'll definitely try sometime during the week.
Awesome, let me know if I can answer any questions!
I happened to be working around that problem. Below is an example workaround, and the comments point out exactly where the problems happen in bokeh code (as far as I managed to figure out). Hope this may be useful.

The relevant part is in the `callback()` function, and is as simple as converting the dataframe to a `dict` of `np.ndarray` objects.

```
import six
from bokeh.application import Application
from bokeh.application.handlers import FunctionHandler
from bokeh.document import Document
from bokeh.models import ColumnDataSource
from bokeh.plotting import Figure

from bokeh.server.server import Server

import pandas as pd
import numpy as np

def modify_doc(doc):
    """

    :param Document doc:
    :return:
    """
    df = pd.DataFrame(
        index=pd.date_range('now', periods=30, freq='T'),
        columns=['A', 'B', 'C'],
        data=np.cumsum(np.random.standard_normal((30,3)), axis=0)
    )

    cds = ColumnDataSource(df)
    figure = Figure(x_axis_type="datetime")
    figure.line("index", "A", source=cds)
    figure.line("index", "B", source=cds)
    figure.line("index", "C", source=cds)

    doc.add_root(figure)

    store = {'df': df}
    def callback():
        # Two problems with bokeh 0.12.6
        #
        # 1. PropertyValueDict._stream checks for np.ndarray but not for pd.Series,
        #    so we need to convert to a dict of np.values ourselves
        #
        # 2. By this point, by some magic, the index has been transformed to milliseconds since epoch
        #    I couldn't figure out where - it happens during serialization but where does that change
        #    get persisted back to the ColumnDataFrame obejct?
        #    Anyway, we need to make that transformation manually...
        #
        df = store['df']
        new_df = pd.DataFrame(
            index=df.index + pd.to_timedelta('30m'),
            columns=df.columns,
            data=np.random.standard_normal((30,3))
        )
        # cds.stream(df)        # This would be nice!

        stream_data = {}
        for k,v in six.iteritems(ColumnDataSource.from_df(new_df)):
            tmp = v if isinstance(v, np.ndarray) else v.values
            if tmp.dtype.kind=='M':
                tmp = tmp.astype('datetime64[us]').astype('int64') / 1000.
            stream_data[k] = tmp

        cds.stream(stream_data)
        store['df'] = new_df

    doc.add_periodic_callback(callback, 1 * 1000)

if __name__ == '__main__':

    bokeh_app = Application(FunctionHandler(modify_doc))
    server = Server({'/': bokeh_app})
    server.run_until_shutdown()
```
@bryevdv Would it make sense to also allow Series streaming - to just add one row to a ColumnDataSource?
@p-himik apologies for any confusion, thats what I was suggesting above, that the streaming code should check for Pandas series and use `Series.append` to implement streaming on series. Though perhaps there is a better way to implement that ads an entire row at once.
@bryevdv Could you please elaborate on your comment in `bokeh/core/property/containers.py`?
```python
        # TODO (bev) Currently this reports old differently for array vs list
        # For arrays is reports the actual old value. For lists, the old value
        # is actually the already updated value. This is because the method
        # self._saved_copy() makes a shallow copy.
```
Below it, we indeed see
```python
L = self[k]
L.extend(new_data[k])
```
which edits the list in-place. Is there anything preventing us from writing `data = self[k] + new_data[k]` and calling `__setitem__` as in the case of `np.ndarray` instead?
If nothing other than the performance considerations are there any consequences of reporting `old` as being the same as `new_data`?
Below is my final solution, that wraps the solution I posted above into a class. It can add any amount of data - not just row at a time. Hope it helps.

One nasty bit is I need to duplicate the serialization code that converts the datetime. It is copied from `bokeh.util.serialization.transform_array` but I only copied a small part of the entire function.

I would suggest the bokeh code that does this be factored our a little so it can be re-used - but if someone copies this for use double check you don't need all of the cases handled in `bokeh.util.serialization.transform_array`

A further complication is the data stored in the `ColumnDataStore` changes between construction and serialization. So you get different behaviour is you `construct -> stream -> connect to document` vs `construct -> connecto to document -> stream`. 

The second case is really the only usual one, but I played with the first one during unit-testing, and was surprised that cases that worked there would fail in production. In particular, the conversion from `datetime64` to `int` only happens after the model is serialised.

```
class PatchedColumnDataSource(ColumnDataSource):
    """Extend ColumnDataSource to better handle Pandas dataframes"""

    # These are undocumented in bokeh, but required to inform bokeh about what to use on the BokehJS side
    # Since we only want to modify some functionality on the python side,
    #  we tell bokeh to use the parent JS object
    # See https://groups.google.com/a/continuum.io/forum/#!topic/bokeh/nbGXAqsE564
    __subtype__ = "PatchedColumnDataSource"
    __view_model__ = "ColumnDataSource"

    def stream(self, new_data, rollover=None, setter=None):
        # Two problems with bokeh 0.12.6
        #
        # 1. PropertyValueDict._stream checks for np.ndarray but not for pd.Series,
        #    so we need to convert to a dict of np.values ourselves
        #
        # 2. By this point, by some magic, the index has been transformed to milliseconds since epoch
        #    I couldn't figure out where - it happens during serialization but where does that change
        #    get persisted back to the ColumnDataFrame obejct?
        #    Anyway, we need to make that transformation manually...
        #
        if isinstance(new_data, pd.DataFrame):
            stream_data = {}
            for k,v in six.iteritems(ColumnDataSource.from_df(new_data)):
                tmp = v if isinstance(v, np.ndarray) else v.values

                if tmp.dtype.kind=='M':
                    tmp = tmp.astype('datetime64[us]').astype('int64') / 1000.    # See bokeh.util.serialization.transform_array
                stream_data[k] = tmp
        else:
            stream_data = new_data

        super(PatchedColumnDataSource, self).stream(stream_data)
```
Ping @bryevdv 
I'm still not sure what the implications of that TODO are. :)
@p-himik sorry I missed the question

> If nothing other than the performance considerations are there any consequences of reporting old as being the same as new_data?

It's an annoying quirk but I also think it's a corner case not often encountered in real practice. Basically, I'd say don't worry about that at all, at least for now. 
@bryevdv But is there any reason not to use `list1 + list2` instead of `list1.extend(list2)`? Why I asking, is that it would not only remove that long `TODO`, but also somewhat simplify the code that I'm writing. If there are no objections against using `list1 + list2`, I'd go with it.
The performance difference is negligible, I think:
```
%timeit list(range(10000)).extend(list(range(10000)))
1000 loops, best of 3: 1.87 ms per loop

%timeit list(range(10000)) + list(range(10000))
100 loops, best of 3: 2.32 ms per loop
```
@p-himik I can't think of any reason *offhand* besides efficiency, but this is also at a very low level so there may be details I am forgetting. A quick smoke test would be to just make the change and see if any tests break. 

That said I think your benchmark is a bit flawed, it's testing the time to execute `list(range(10000))` every time, which is considerable. I looked at this:

```
In [7]: x = list(range(10000))

In [8]: y = list(range(10000))

In [9]: %timeit x + y
10000 loops, best of 3: 65.5 µs per loop

In [10]: %timeit list(range(10000)).extend(list(range(10000)))
1000 loops, best of 3: 352 µs per loop

In [11]: %timeit list(range(10000)).extend(y)
10000 loops, best of 3: 195 µs per loop
``` 
Under the assumption that the difference in the last two, 352 - 195 = 157, accounts for one of the list creations, then we can approximate 195 - 157 = 38us as the actual time difference for just the `extend` operation. That's almost a 2x improvement over the `+` version, which I would say is not insignificant. There are folks using Bokeh in very intensive applications and are already asking to find places to reduce overhead. I think my inclination is to keep things as fast as possible in this one code path. 

Edit: 

actually just timing the list creation bears out the approximation above:
```
In [12]: %timeit list(range(10000))
10000 loops, best of 3: 162 µs per loop
```

Resolving patch:
diff --git a/bokeh/core/property/containers.py b/bokeh/core/property/containers.py
--- a/bokeh/core/property/containers.py
+++ b/bokeh/core/property/containers.py
@@ -45,6 +45,10 @@ class SomeModel(Model):
 
 '''
 from __future__ import absolute_import, print_function
+from ...util.dependencies import import_optional
+
+pd = import_optional('pandas')
+
 
 def notify_owner(func):
     ''' A decorator for mutating methods of property container classes
@@ -320,11 +324,17 @@ def _stream(self, doc, source, new_data, rollover=None, setter=None):
 
         import numpy as np
 
+        # pandas/issues/13918
+        if pd and isinstance(new_data, pd.DataFrame):
+            new_items = new_data.iteritems()
+        else:
+            new_items = new_data.items()
+
         # TODO (bev) Currently this reports old differently for array vs list
         # For arrays is reports the actual old value. For lists, the old value
         # is actually the already updated value. This is because the method
         # self._saved_copy() makes a shallow copy.
-        for k, v in new_data.items():
+        for k, v in new_items:
             if isinstance(self[k], np.ndarray):
                 data = np.append(self[k], new_data[k])
                 if rollover and len(data) > rollover:
diff --git a/bokeh/models/sources.py b/bokeh/models/sources.py
--- a/bokeh/models/sources.py
+++ b/bokeh/models/sources.py
@@ -172,7 +172,7 @@ def _data_from_df(df):
         '''
         _df = df.copy()
         index = _df.index
-        tmp_data = _df.to_dict('series')
+        tmp_data = {c: v.values for c, v in _df.iteritems()}
 
         new_data = {}
         for k, v in tmp_data.items():
@@ -351,8 +351,10 @@ def _stream(self, new_data, rollover=None, setter=None):
         entire data set to be re-sent.
 
         Args:
-            new_data (dict[str, seq]) : a mapping of column names to sequences of
-                new data to append to each column.
+            new_data (dict[str, seq] or DataFrame or Series) : a mapping of
+                column names to sequences of new data to append to each column,
+                a pandas DataFrame, or a pandas Series in case of a single row -
+                in this case the Series index is used as column names
 
                 All columns of the data source must be present in ``new_data``,
                 with identical-length append data.
@@ -390,9 +392,12 @@ def _stream(self, new_data, rollover=None, setter=None):
             source.stream(new_data)
 
         '''
-        import numpy as np
-
-        newkeys = set(new_data.keys())
+        if pd and isinstance(new_data, pd.Series):
+            new_data = new_data.to_frame().T
+        if pd and isinstance(new_data, pd.DataFrame):
+            newkeys = set(new_data.columns)
+        else:
+            newkeys = set(new_data.keys())
         oldkeys = set(self.data.keys())
         if newkeys != oldkeys:
             missing = oldkeys - newkeys
@@ -406,17 +411,21 @@ def _stream(self, new_data, rollover=None, setter=None):
             else:
                 raise ValueError("Must stream updates to all existing columns (extra: %s)" % ", ".join(sorted(extra)))
 
-        lengths = set()
-        for x in new_data.values():
-            if isinstance(x, np.ndarray):
-                if len(x.shape) != 1:
-                    raise ValueError("stream(...) only supports 1d sequences, got ndarray with size %r" % (x.shape,))
-                lengths.add(x.shape[0])
-            else:
-                lengths.add(len(x))
+        if not (pd and isinstance(new_data, pd.DataFrame)):
+            import numpy as np
+
+            lengths = set()
+            arr_types = (np.ndarray, pd.Series) if pd else np.ndarray
+            for k, x in new_data.items():
+                if isinstance(x, arr_types):
+                    if len(x.shape) != 1:
+                        raise ValueError("stream(...) only supports 1d sequences, got ndarray with size %r" % (x.shape,))
+                    lengths.add(x.shape[0])
+                else:
+                    lengths.add(len(x))
 
-        if len(lengths) > 1:
-            raise ValueError("All streaming column updates must be the same length")
+            if len(lengths) > 1:
+                raise ValueError("All streaming column updates must be the same length")
 
         self.data._stream(self.document, self, new_data, rollover, setter)
 


Test patch:
diff --git a/bokeh/models/tests/test_sources.py b/bokeh/models/tests/test_sources.py
--- a/bokeh/models/tests/test_sources.py
+++ b/bokeh/models/tests/test_sources.py
@@ -40,7 +40,7 @@ def test_init_dataframe_arg(self):
         ds = ColumnDataSource(df)
         self.assertTrue(set(df.columns).issubset(set(ds.column_names)))
         for key in data.keys():
-            self.assertIsInstance(ds.data[key], pd.Series)
+            self.assertIsInstance(ds.data[key], np.ndarray)
             self.assertEquals(list(df[key]), list(ds.data[key]))
         self.assertIsInstance(ds.data['index'], np.ndarray)
         self.assertEquals([0, 1], list(ds.data['index']))
@@ -53,7 +53,7 @@ def test_init_dataframe_data_kwarg(self):
         ds = ColumnDataSource(data=df)
         self.assertTrue(set(df.columns).issubset(set(ds.column_names)))
         for key in data.keys():
-            self.assertIsInstance(ds.data[key], pd.Series)
+            self.assertIsInstance(ds.data[key], np.ndarray)
             self.assertEquals(list(df[key]), list(ds.data[key]))
         self.assertIsInstance(ds.data['index'], np.ndarray)
         self.assertEquals([0, 1], list(ds.data['index']))
@@ -70,7 +70,7 @@ def test_init_groupby_arg(self):
         self.assertIsInstance(ds.data['origin_cyl'], np.ndarray)
         for key in s.columns.values:
             k2 = "_".join(key)
-            self.assertIsInstance(ds.data[k2], pd.Series)
+            self.assertIsInstance(ds.data[k2], np.ndarray)
             self.assertEquals(list(s[key]), list(ds.data[k2]))
 
     @skipIf(not is_pandas, "pandas not installed")
@@ -84,7 +84,7 @@ def test_init_groupby_data_kwarg(self):
         self.assertIsInstance(ds.data['origin_cyl'], np.ndarray)
         for key in s.columns.values:
             k2 = "_".join(key)
-            self.assertIsInstance(ds.data[k2], pd.Series)
+            self.assertIsInstance(ds.data[k2], np.ndarray)
             self.assertEquals(list(s[key]), list(ds.data[k2]))
 
     @skipIf(not is_pandas, "pandas not installed")
@@ -97,7 +97,7 @@ def test_init_groupby_with_None_subindex_name(self):
         self.assertIsInstance(ds.data['index'], np.ndarray)
         for key in s.columns.values:
             k2 = "_".join(key)
-            self.assertIsInstance(ds.data[k2], pd.Series)
+            self.assertIsInstance(ds.data[k2], np.ndarray)
             self.assertEquals(list(s[key]), list(ds.data[k2]))
 
     def test_add_with_name(self):
@@ -190,6 +190,170 @@ def mock(*args, **kw):
         self.assertEqual(stuff['args'], ("doc", ds, dict(a=[11, 12], b=[21, 22]), "foo", None))
         self.assertEqual(stuff['kw'], {})
 
+    def _assert_equal_dicts_of_arrays(self, d1, d2):
+        self.assertEqual(d1.keys(), d2.keys())
+        for k, v in d1.items():
+            self.assertEqual(type(v), type(d2[k]))
+            self.assertTrue(np.array_equal(v, d2[k]))
+
+    @skipIf(not is_pandas, "pandas not installed")
+    def test_stream_dict_to_ds_created_from_df(self):
+        data = pd.DataFrame(dict(a=[10], b=[20], c=[30])).set_index('c')
+        ds = ColumnDataSource(data)
+        ds._document = "doc"
+
+        notify_owners_stuff = {}
+
+        def notify_owners_mock(*args, **kw):
+            notify_owners_stuff['args'] = args
+            notify_owners_stuff['kw'] = kw
+        ds.data._notify_owners = notify_owners_mock
+
+        stream_stuff = {}
+        data_stream = ds.data._stream
+
+        def stream_wrapper(*args, **kwargs):
+            stream_stuff['args'] = args
+            stream_stuff['kwargs'] = kwargs
+            data_stream(*args, **kwargs)
+        ds.data._stream = stream_wrapper
+
+        ds._stream(dict(a=[11, 12],
+                        b=np.array([21, 22]),
+                        c=pd.Series([31, 32])), 7)
+
+        self.assertEqual(len(stream_stuff['args']), 5)
+        expected_stream_args = ("doc", ds, dict(a=[11, 12],
+                                                b=np.array([21, 22]),
+                                                c=pd.Series([31, 32])), 7, None)
+        for i, (arg, ex_arg) in enumerate(zip(stream_stuff['args'],
+                                              expected_stream_args)):
+            if i == 2:
+                self.assertEqual(arg['a'], ex_arg['a'])
+                del arg['a'], ex_arg['a']
+                self._assert_equal_dicts_of_arrays(arg, ex_arg)
+            else:
+                self.assertEqual(arg, ex_arg)
+
+        self.assertEqual(stream_stuff['kwargs'], {})
+
+        self.assertEqual(len(notify_owners_stuff['args']), 1)
+        self._assert_equal_dicts_of_arrays(notify_owners_stuff['args'][0],
+                                           dict(a=np.array([10]),
+                                                b=np.array([20]),
+                                                c=np.array([30])))
+
+        self._assert_equal_dicts_of_arrays(dict(ds.data),
+                                           dict(a=np.array([10, 11, 12]),
+                                                b=np.array([20, 21, 22]),
+                                                c=np.array([30, 31, 32])))
+
+    @skipIf(not is_pandas, "pandas not installed")
+    def test_stream_series_to_ds_created_from_df(self):
+        data = pd.DataFrame(dict(a=[10], b=[20], c=[30])).set_index('c')
+        ds = ColumnDataSource(data)
+        ds._document = "doc"
+
+        notify_owners_stuff = {}
+
+        def notify_owners_mock(*args, **kw):
+            notify_owners_stuff['args'] = args
+            notify_owners_stuff['kw'] = kw
+
+        ds.data._notify_owners = notify_owners_mock
+
+        stream_stuff = {}
+        data_stream = ds.data._stream
+
+        def stream_wrapper(*args, **kwargs):
+            stream_stuff['args'] = args
+            stream_stuff['kwargs'] = kwargs
+            data_stream(*args, **kwargs)
+
+        ds.data._stream = stream_wrapper
+
+        ds._stream(pd.Series([11, 21, 31], index=list('abc')), 7)
+
+        self.assertEqual(len(stream_stuff['args']), 5)
+        expected_stream_args = ("doc", ds, dict(a=np.array([11]),
+                                                b=np.array([21]),
+                                                c=np.array([31])), 7, None)
+        for i, (arg, ex_arg) in enumerate(zip(stream_stuff['args'],
+                                              expected_stream_args)):
+            if i == 2:
+                arg = {k: v.values for k, v in arg.to_dict('series').items()}
+                self._assert_equal_dicts_of_arrays(arg, ex_arg)
+            else:
+                self.assertEqual(arg, ex_arg)
+
+        self.assertEqual(stream_stuff['kwargs'], {})
+
+        self.assertEqual(len(notify_owners_stuff['args']), 1)
+        self._assert_equal_dicts_of_arrays(notify_owners_stuff['args'][0],
+                                           dict(a=np.array([10]),
+                                                b=np.array([20]),
+                                                c=np.array([30])))
+
+        self._assert_equal_dicts_of_arrays(dict(ds.data),
+                                           dict(a=np.array([10, 11]),
+                                                b=np.array([20, 21]),
+                                                c=np.array([30, 31])))
+
+    @skipIf(not is_pandas, "pandas not installed")
+    def test_stream_df_to_ds_created_from_df(self):
+        data = pd.DataFrame(dict(a=[10], b=[20], c=[30])).set_index('c')
+        ds = ColumnDataSource(data)
+        ds._document = "doc"
+
+        notify_owners_stuff = {}
+
+        def notify_owners_mock(*args, **kw):
+            notify_owners_stuff['args'] = args
+            notify_owners_stuff['kw'] = kw
+
+        ds.data._notify_owners = notify_owners_mock
+
+        stream_stuff = {}
+        data_stream = ds.data._stream
+
+        def stream_wrapper(*args, **kwargs):
+            stream_stuff['args'] = args
+            stream_stuff['kwargs'] = kwargs
+            data_stream(*args, **kwargs)
+
+        ds.data._stream = stream_wrapper
+
+        ds._stream(pd.DataFrame(dict(a=[11, 12],
+                                     b=[21, 22],
+                                     c=[31, 32])), 7)
+
+        self.assertEqual(len(stream_stuff['args']), 5)
+        expected_steam_data = dict(a=np.array([11, 12]),
+                                   b=np.array([21, 22]),
+                                   c=np.array([31, 32]))
+        expected_args = ("doc", ds, expected_steam_data, 7, None)
+        for i, (arg, ex_arg) in enumerate(zip(stream_stuff['args'], expected_args)):
+            if i == 2:
+                arg = {k: v.values for k, v in arg.to_dict('series').items()}
+                self.assertEqual(arg.keys(), ex_arg.keys())
+                for k, v in arg.items():
+                    self.assertTrue(np.array_equal(v, ex_arg[k]))
+            else:
+                self.assertEqual(stream_stuff['args'][i], expected_args[i])
+
+        self.assertEqual(stream_stuff['kwargs'], {})
+
+        self.assertEqual(len(notify_owners_stuff['args']), 1)
+        self._assert_equal_dicts_of_arrays(notify_owners_stuff['args'][0],
+                                           dict(a=np.array([10]),
+                                                b=np.array([20]),
+                                                c=np.array([30])))
+
+        self._assert_equal_dicts_of_arrays(dict(ds.data),
+                                           dict(a=np.array([10, 11, 12]),
+                                                b=np.array([20, 21, 22]),
+                                                c=np.array([30, 31, 32])))
+
     def test_patch_bad_columns(self):
         ds = ColumnDataSource(data=dict(a=[10, 11], b=[20, 21]))
         with self.assertRaises(ValueError) as cm: