Repository: bokeh/bokeh

Issue:
Task: Add user warn/error to ColumnDataSource.from_df method about DataFrame containing "index" column
There's some weirdness around using pandas DataFrames to create ColumnDataSources for graph visualization in Bokeh. Currently Bokeh requires the GraphRenderer.node_renderer.data_source to have a column named "index". This column is the reference for the edge's start and end values.

The issue is that creating a ColumnDataSource with a pd.DataFrame with:
* an unnamed Index
* a column named "index"

The resulting CDS will have a "index" column containing the dataframe (capital-i) Index and not the column. This is a gotcha in pandas where ``df.index`` refers to the Index but ``df["index"]`` refers to a column named "index".

Bokeh 0.12.7dev16
Task: Add user warn/error to ColumnDataSource.from_df method about DataFrame containing "index" column
There's some weirdness around using pandas DataFrames to create ColumnDataSources for graph visualization in Bokeh. Currently Bokeh requires the GraphRenderer.node_renderer.data_source to have a column named "index". This column is the reference for the edge's start and end values.

The issue is that creating a ColumnDataSource with a pd.DataFrame with:
* an unnamed Index
* a column named "index"

The resulting CDS will have a "index" column containing the dataframe (capital-i) Index and not the column. This is a gotcha in pandas where ``df.index`` refers to the Index but ``df["index"]`` refers to a column named "index".

Bokeh 0.12.7dev16


Discussion:
Thinking about it some more, this is a non graph-specific issue related to the ColumnDataSource._from_df method. I think the best behavior would be to warn/error users about the ambiguity in such a situation.
Agree with the warning. The issue is easy to mitigate once identified, and a little nudge in the right direction is always helpful.  
MRE:

```
In [1]: import pandas as pd

In [2]: df = pd.DataFrame(dict(index=[0.0, 1.1, 2.2]))

In [3]: from bokeh.models import ColumnDataSource

In [4]: s = ColumnDataSource(df)

In [5]: s.data
Out[5]: {'index': array([0, 1, 2])}
```
Thinking about it some more, this is a non graph-specific issue related to the ColumnDataSource._from_df method. I think the best behavior would be to warn/error users about the ambiguity in such a situation.
Agree with the warning. The issue is easy to mitigate once identified, and a little nudge in the right direction is always helpful.  
MRE:

```
In [1]: import pandas as pd

In [2]: df = pd.DataFrame(dict(index=[0.0, 1.1, 2.2]))

In [3]: from bokeh.models import ColumnDataSource

In [4]: s = ColumnDataSource(df)

In [5]: s.data
Out[5]: {'index': array([0, 1, 2])}
```

Resolving patch:
diff --git a/bokeh/models/sources.py b/bokeh/models/sources.py
--- a/bokeh/models/sources.py
+++ b/bokeh/models/sources.py
@@ -65,10 +65,15 @@ class ColumnDataSource(ColumnarDataSource):
           source = ColumnDataSource(df)
 
       In this case the CDS will have columns corresponding to the columns of
-      the ``DataFrame``. If the ``DataFrame`` has a named index column, then
-      CDS will also have a column with this name. However, if the index name
-      (or any subname of a ``MultiIndex``) is ``None``, then the CDS will have
-      a column generically named ``index`` for the index.
+      the ``DataFrame``. If the ``DataFrame`` columns have multiple levels,
+      they will be flattend using an underscore (e.g. level_0_col_level_1_col).
+      The index of the DataFrame will be flattened to an ``Index`` of tuples
+      if it's a ``MultiIndex``, and then reset using ``reset_index``. The result
+      will be a column with the same name if the index was named, or
+      level_0_name_level_1_name if it was a named ``MultiIndex``. If the
+      ``Index`` did not have a name or the ``MultiIndex`` name could not be
+      flattened/determined, the ``reset_index`` function will name the index column
+      ``index``, or ``level_0`` if the name ``index`` is not available.
 
     * A Pandas ``GroupBy`` object
 
@@ -150,16 +155,28 @@ def _data_from_df(df):
 
         '''
         _df = df.copy()
+
+        # Flatten columns
+        if isinstance(df.columns, pd.MultiIndex):
+            try:
+                _df.columns = ['_'.join(col) for col in _df.columns.values]
+            except TypeError:
+                raise TypeError('Could not flatten MultiIndex columns. '
+                                'use string column names or flatten manually')
+        # Flatten index
+        index_name = ColumnDataSource._df_index_name(df)
+        if index_name == 'index':
+            _df.index = pd.Index(_df.index.values)
+        else:
+            _df.index = pd.Index(_df.index.values, name=index_name)
+        _df.reset_index(inplace=True)
+
         tmp_data = {c: v.values for c, v in _df.iteritems()}
 
         new_data = {}
         for k, v in tmp_data.items():
-            if isinstance(k, tuple):
-                k = "_".join(k)
             new_data[k] = v
 
-        index_name = ColumnDataSource._df_index_name(df)
-        new_data[index_name] = _df.index.values
         return new_data
 
     @staticmethod


Test patch:
diff --git a/bokeh/models/tests/test_sources.py b/bokeh/models/tests/test_sources.py
--- a/bokeh/models/tests/test_sources.py
+++ b/bokeh/models/tests/test_sources.py
@@ -53,6 +53,30 @@ def test_init_dataframe_data_kwarg(self, pd):
         assert [0, 1] == list(ds.data['index'])
         assert set(ds.column_names) - set(df.columns) == set(["index"])
 
+    def test_init_dataframe_index_named_column(self, pd):
+        data = dict(a=[1, 2], b=[2, 3], index=[4, 5])
+        df = pd.DataFrame(data)
+        ds = ColumnDataSource(data=df)
+        assert set(df.columns).issubset(set(ds.column_names))
+        for key in data.keys():
+            assert isinstance(ds.data[key], np.ndarray)
+            assert list(df[key]) == list(ds.data[key])
+        assert isinstance(ds.data['level_0'], np.ndarray)
+        assert [0, 1] == list(ds.data['level_0'])
+        assert set(ds.column_names) - set(df.columns) == set(["level_0"])
+
+    def test_init_dataframe_nonstring_named_column(self, pd):
+        data = {1: [1, 2], 2: [2, 3]}
+        df = pd.DataFrame(data)
+        with pytest.raises(ValueError, match=r'expected an element of.*'):
+            ColumnDataSource(data=df)
+
+    def test_init_dataframe_nonstring_named_multicolumn(self, pd):
+        data = {(1, 2): [1, 2], (2, 3): [2, 3]}
+        df = pd.DataFrame(data)
+        with pytest.raises(TypeError, match=r'Could not flatten.*'):
+            ColumnDataSource(data=df)
+
     def test_init_groupby_arg(self, pd):
         from bokeh.sampledata.autompg import autompg as df
         group = df.groupby(by=['origin', 'cyl'])