Repository: bokeh/bokeh

Issue:
[Feature Request] Option to collapse DataTable rows
Please refer to [this thread](https://groups.google.com/a/continuum.io/forum/#!topic/bokeh/AgWWxSUJuzQ) on Google Groups.

It would be nice to have this feature for tree data structure.

Thanks.



Discussion:
I don't really understand the use case or requirements. In particular, existing Bokeh data sources are, to my mind anyway, fundamentally not hierarchical. So what exactly does it mean to adapt them into a collapsible (i.e. tree-style) layout? Alternatively are you proposing an entirely *new* data structure? 

So for there to be any chance of movement on this, someone will have to put together a *much* more thorough proposal:

* some write-ups of use cases where this would be useful and why
* explicit proposed new Bokeh API / syntax that users will have
* detailed descriptions of desired behaviours
* example scripts that would demonstrate this feature, once implemented

@lgasnier are you able to work up a real proposal? If not we will have to close this as too underspecified. 
Sorry, my request is too brief and not obvious.

**Use case**

My concrete use case is the following. I have an ecological status that depends on:

- benthic invertebrate fauna
- aquatic flora
- fish fauna
- …
- chemical and physico-chemical elements :
  * oxygenation conditions
  * temperature
  * nutrients conditions
  * ...
- …

See screenshot below (sorry, example in french)
![image](https://user-images.githubusercontent.com/25739150/43155934-0dc51142-8f79-11e8-9a07-a6fe814b7d3f.png)

With this tree data structure, I made a `Div` widget. Now, I would like to add in the layout a plot of the data used to calculate the status of each element (for example, nitrogen vs time). A click on one element of this tree would update the figure. But `Div` is just a display widget and can’t have a callback to interact with such a plot.

When I saw the thread mentioned above, I thought DataTable could be the right widget for my use case with some enhancements. But I don’t know if it represents a lot of work ;-)

So below is what I came up with.

**New attributes proposed to TableColumn class**

- *accordion* (boolean): if `True`, field must be a tuple which size is the number of levels in the tree. See example below. A branch has a tuple with `None` as last index or indices according to its level, Note that a child must have a parent i.e. if tuple[i] is `None` then tuple[j] must be `None` for j > i.
- *expand_field* (string; optional): the name of the source field used to initiate the expanded/collapsed rows. The field must be a list of booleans. Default `True` for all rows. For the leaves of the tree, `True` or `False` doesn’t matter.
- *offset* (pixels ? number of characters ?) to shift column content according to the tree level.
- *expand_mark* and *collapse_mark* (character ? image ?)

**Desired behavior**

If a column has children (ie, tuple[-1] is None), add a mark to manage accordion (to collapse/expand the rows of the datatable). To collapse children rows of a parent, these children rows would be filtered according to the tuple content of the column values.

**Dummy script example**

```python
from bokeh.io import output_file, show
from bokeh.layouts import widgetbox
from bokeh.models import ColumnDataSource
from bokeh.models.widgets import DataTable, DateFormatter, TableColumn

output_file("data_table.html")

data = dict(
        name=[("John", None, None),
              ("John", "Thomas", None),
              ("John", "Thomas", "Joe"),
              ("John", "Thomas", "Ken"),
              ("John", "James", None),
              ("John", "James", "Pat"),
              ("George", None, None),
              ("George", "Martin", None),
              ("George", "Martin", "Emily"),
              ("George", "Andrew", None),
              ("George", "Andrew", "Ron")],
        age=[70, 40, 10, 15, 30, 8, 65, 35, 15, 40, 18],
        expanded=[True, True, False, False, False,
                  False, True, True, True, True, True],
    )
source = ColumnDataSource(data)

columns = [
        TableColumn(field="name",
                   title="Children, fathers and grandfathers",
                   accordion=True,
                   expand_field="expanded",
                   offset="10px",
                   expand_mark="►",
                   collapse_mark="▼"),
        TableColumn(field="age", title="Age"),
    ]
data_table = DataTable(source=source, columns=columns, width=400, height=280)

show(widgetbox(data_table))
```

*Expected result*:

Children, fathers and grandfathers | Age
-- | --
▼John | 70
...▼Thomas | 40
......Joe | 10
......Ken | 15
...►James | 30
▼George | 65
...▼Martin | 35
......Emily | 15
...▼Andrew | 40
......Ron | 18

Is it enough to take into account this feature request? Do you need some more descriptions?

Thanks.

Thanks for the additional details @lgasnier With that detail I can see that something like that might be possible and useful. 

But I should be up front about expectations re: timing. This is probably not something that can be addressed by the small core team in the very near future. The DataTable is already one of the most complicated widgets, and it has several current regressions due to a lapse in our automated testing. We need to get it back in reasonable shape before adding more complexity to it. I am working on fixing our testing infra which is itself a large task, then going to get the current DataTable stable under test. The other thread of work now is a final clean up of layout. Those are the priorities for an upcoming 1.0 release. Since this issue describes a new feature without breaking API, it definitely won't come before the 1.0 which is a few months away. 
First, great thanks to bokeh team.

I understand these priorities. In the meantime, I'll try to use select widgets to filter the DataTable by level.

Resolving patch:
diff --git a/bokeh/models/widgets/tables.py b/bokeh/models/widgets/tables.py
--- a/bokeh/models/widgets/tables.py
+++ b/bokeh/models/widgets/tables.py
@@ -39,25 +39,31 @@
 #-----------------------------------------------------------------------------
 
 __all__ = (
+    'AvgAggregator',
     'BooleanFormatter',
     'CellFormatter',
     'CellEditor',
     'CheckboxEditor',
+    'DataCube',
     'DataTable',
     'DateEditor',
     'DateFormatter',
+    'GroupingInfo',
     'HTMLTemplateFormatter',
     'IntEditor',
+    'MaxAggregator',
+    'MinAggregator',
     'NumberEditor',
     'NumberFormatter',
     'PercentEditor',
     'SelectEditor',
     'StringEditor',
     'StringFormatter',
+    'SumAggregator',
     'TableColumn',
+    'TableWidget',
     'TextEditor',
     'TimeEditor',
-    'TableWidget',
 )
 
 #-----------------------------------------------------------------------------
@@ -76,6 +82,15 @@ class CellEditor(Model):
 
     '''
 
+@abstract
+class RowAggregator(Model):
+    ''' Abstract base class for data cube's row formatters.
+
+    '''
+    field_ = String('', help="""
+    Refers to the table column being aggregated
+    """)
+
 #-----------------------------------------------------------------------------
 # General API
 #-----------------------------------------------------------------------------
@@ -499,6 +514,26 @@ class DateEditor(CellEditor):
 
     '''
 
+class AvgAggregator(RowAggregator):
+    ''' Simple average across multiple rows.
+
+    '''
+
+class MinAggregator(RowAggregator):
+    ''' Smallest value across multiple rows.
+
+    '''
+
+class MaxAggregator(RowAggregator):
+    ''' Largest value across multiple rows.
+
+    '''
+
+class SumAggregator(RowAggregator):
+    ''' Simple sum across multiple rows.
+
+    '''
+
 class TableColumn(Model):
     ''' Table column widget.
 
@@ -639,6 +674,35 @@ class DataTable(TableWidget):
     The height of each row in pixels.
     """)
 
+class GroupingInfo(Model):
+    '''Describes how to calculate totals and sub-totals
+    '''
+
+    getter = String('', help="""
+    References the column which generates the unique keys of this sub-total (groupby).
+    """)
+
+    aggregators = List(Instance(RowAggregator), help="""
+    Describes how to aggregate the columns which will populate this sub-total.
+    """)
+
+    collapsed = Bool(False, help="""
+    Whether the corresponding sub-total is expanded or collapsed by default.
+    """)
+
+class DataCube(DataTable):
+    '''Specialized DataTable with collapsing groups, totals, and sub-totals.
+    '''
+
+    grouping = List(Instance(GroupingInfo), help="""
+    Describe what aggregation operations used to define sub-totals and totals
+    """)
+
+    target = Instance(DataSource, help="""
+    Two column datasource (row_indices & labels) describing which rows of the
+    data cubes are expanded or collapsed
+    """)
+
 #-----------------------------------------------------------------------------
 # Private API
 #-----------------------------------------------------------------------------
diff --git a/examples/models/file/data_cube.py b/examples/models/file/data_cube.py
new file mode 100644
--- /dev/null
+++ b/examples/models/file/data_cube.py
@@ -0,0 +1,40 @@
+from bokeh.document import Document
+from bokeh.embed import file_html
+from bokeh.models import ColumnDataSource
+from bokeh.models.widgets import StringFormatter, TableColumn, GroupingInfo, SumAggregator, DataCube
+from bokeh.resources import INLINE
+from bokeh.util.browser import view
+
+source = ColumnDataSource(data=dict(
+    d0=['A', 'E', 'E', 'E', 'J', 'L', 'M'],
+    d1=['B', 'D', 'D', 'H', 'K', 'L', 'N'],
+    d2=['C', 'F', 'G', 'H', 'K', 'L', 'O'],
+    px=[10, 20, 30, 40, 50, 60, 70],
+))
+
+target = ColumnDataSource(data=dict(row_indices=[], labels=[]))
+
+formatter = StringFormatter(font_style='bold')
+
+columns = [
+    TableColumn(field='d2', title='Name', width=80, sortable=False, formatter=formatter),
+    TableColumn(field='px', title='Price', width=40, sortable=False),
+]
+
+grouping = [
+    GroupingInfo(getter='d0', aggregators=[SumAggregator(field_='px')]),
+    GroupingInfo(getter='d1', aggregators=[SumAggregator(field_='px')])
+]
+
+cube = DataCube(source=source, columns=columns, grouping=grouping, target=target)
+
+doc = Document()
+doc.add_root(cube)
+
+if __name__ == '__main__':
+    doc.validate()
+    filename = "data_cube.html"
+    with open(filename, "w") as f:
+        f.write(file_html(doc, INLINE, "Demonstration of Data Cube with Aggregations"))
+    print("Wrote", filename)
+    view(filename)


Test patch:
diff --git a/bokehjs/test/models/widgets/tables/data_cube.ts b/bokehjs/test/models/widgets/tables/data_cube.ts
new file mode 100644
--- /dev/null
+++ b/bokehjs/test/models/widgets/tables/data_cube.ts
@@ -0,0 +1,77 @@
+import { expect } from "chai"
+
+import { ColumnDataSource } from "@bokehjs/models/sources/column_data_source"
+import { CDSView } from "@bokehjs/models/sources/cds_view"
+import { TableColumn } from "@bokehjs/models/widgets/tables/table_column"
+
+import { GroupingInfo, DataCubeProvider, DataCube } from "@bokehjs/models/widgets/tables/data_cube"
+import { SumAggregator } from "@bokehjs/models/widgets/tables/row_aggregators"
+
+describe("data_cube module", function() {
+
+  describe("DataCube class", function() {
+    it("DataCube constructs", function() {
+      const dataCube = new DataCube(this.attrs)
+      expect(dataCube).is.not.null
+    })
+  })
+
+  describe("DataCubeProvider class", function() {
+    before("setup a new datacube", function() {
+      this.source = new ColumnDataSource({
+        data: {
+          color: ['red', 'red', 'red', 'green', 'green', 'blue'],
+          width: ['wide', 'wide', 'narrow', 'wide', 'narrow', 'wide'],
+          value: [10, 20, 30, 40, 50, 60],
+        },
+      })
+      this.view = new CDSView({ source: this.source })
+      this.columns = [
+        new TableColumn({ field: 'color' }),
+        new TableColumn({ field: 'width' }),
+        new TableColumn({ field: 'value' }),
+      ]
+
+      const aggregators = [new SumAggregator({ field_: 'value' })]
+      this.grouping = [
+        new GroupingInfo({ getter: 'color', aggregators, collapsed: true }),
+        new GroupingInfo({ getter: 'width', aggregators, collapsed: true }),
+      ]
+    })
+
+    it("DataCube groups as expected", function() {
+      const target = new ColumnDataSource({ data: { row_indices: [], labels: [] } })
+      const provider = new DataCubeProvider(this.source, this.view, this.columns, target)
+      provider.setGrouping(this.grouping)
+      expect(target.data.row_indices).to.deep.equal([[5], [3, 4], [0, 1, 2]])
+    })
+
+    it("Expanding modifies groups", function() {
+      const target = new ColumnDataSource({ data: { row_indices: [], labels: [] } })
+      const provider = new DataCubeProvider(this.source, this.view, this.columns, target)
+      provider.setGrouping(this.grouping)
+      provider.expandGroup("red")
+      provider.refresh()
+      expect(target.data.row_indices).to.deep.equal([[5], [3, 4], [0, 1, 2], [2], [0, 1]])
+      provider.expandGroup("red:|:wide")
+      provider.refresh()
+      expect(target.data.row_indices).to.deep.equal([[5], [3, 4], [0, 1, 2], [2], [0, 1], 0, 1])
+    })
+
+    it("Collapsing inverts expanding", function() {
+      const target = new ColumnDataSource({ data: { row_indices: [], labels: [] } })
+      const provider = new DataCubeProvider(this.source, this.view, this.columns, target)
+      provider.setGrouping(this.grouping)
+      provider.expandGroup("red")
+      provider.expandGroup("red:|:wide")
+      provider.refresh()
+      expect(target.data.row_indices).to.deep.equal([[5], [3, 4], [0, 1, 2], [2], [0, 1], 0, 1])
+      provider.collapseGroup("red:|:wide")
+      provider.refresh()
+      expect(target.data.row_indices).to.deep.equal([[5], [3, 4], [0, 1, 2], [2], [0, 1]])
+      provider.collapseGroup("red")
+      provider.refresh()
+      expect(target.data.row_indices).to.deep.equal([[5], [3, 4], [0, 1, 2]])
+    })
+  })
+})
diff --git a/bokehjs/test/models/widgets/tables/data_table.ts b/bokehjs/test/models/widgets/tables/data_table.ts
--- a/bokehjs/test/models/widgets/tables/data_table.ts
+++ b/bokehjs/test/models/widgets/tables/data_table.ts
@@ -3,7 +3,7 @@ import {expect} from "chai"
 import {ColumnDataSource} from "@bokehjs/models/sources/column_data_source"
 import {CDSView} from "@bokehjs/models/sources/cds_view"
 
-import {DataProvider, DataTable, DTINDEX_NAME} from "@bokehjs/models/widgets/tables/data_table"
+import {TableDataProvider, DataTable, DTINDEX_NAME} from "@bokehjs/models/widgets/tables/data_table"
 
 import {range} from "@bokehjs/core/util/array"
 
@@ -50,27 +50,27 @@ describe("data_table module", () => {
     it("should raise an error if DTINDEX_NAME is in source", () => {
       const bad = new ColumnDataSource({data: {__bkdt_internal_index__: [0,1,2,10], bar: [3.4, 1.2, 0, -10]}})
       const view = new CDSView({source: bad})
-      expect(() => new DataProvider(bad, view)).to.throw(Error)
+      expect(() => new TableDataProvider(bad, view)).to.throw(Error)
     })
 
     it("should construct an internal index", () => {
       const source = new ColumnDataSource({data: {index: [0,1,2,10], bar: [3.4, 1.2, 0, -10]}})
       const view = new CDSView({source})
-      const dp = new DataProvider(source, view)
+      const dp = new TableDataProvider(source, view)
       expect(dp.index).to.deep.equal([0,1,2,3])
     })
 
     it("should report the data source length", () => {
       const source = new ColumnDataSource({data: {index: [0,1,2,10], bar: [3.4, 1.2, 0, -10]}})
       const view = new CDSView({source})
-      const dp = new DataProvider(source, view)
+      const dp = new TableDataProvider(source, view)
       expect(dp.getLength()).to.equal(4)
     })
 
     it("should return items when unsorted", () => {
       const source = new ColumnDataSource({data: {index: [0,1,2,10], bar: [3.4, 1.2, 0, -10]}})
       const view = new CDSView({source})
-      const dp = new DataProvider(source, view)
+      const dp = new TableDataProvider(source, view)
 
       expect(dp.getItem(0)).to.deep.equal({__bkdt_internal_index__: 0, index: 0,  bar: 3.4})
       expect(dp.getItem(1)).to.deep.equal({__bkdt_internal_index__: 1, index: 1,  bar: 1.2})
@@ -81,7 +81,7 @@ describe("data_table module", () => {
     it("should return items when sorted", () => {
       const source = new ColumnDataSource({data: {index: [0,1,2,10], bar: [3.4, 1.2, 0, -10]}})
       const view = new CDSView({source})
-      const dp = new DataProvider(source, view)
+      const dp = new TableDataProvider(source, view)
       const fake_col = {sortAsc: true, sortCol: {field: "bar"}}
       dp.sort([fake_col])
 
@@ -94,7 +94,7 @@ describe("data_table module", () => {
     it("should return fields when unsorted", () => {
       const source = new ColumnDataSource({data: {index: [0,1,2,10], bar: [3.4, 1.2, 0, -10]}})
       const view = new CDSView({source})
-      const dp = new DataProvider(source, view)
+      const dp = new TableDataProvider(source, view)
 
       expect(dp.getField(0, "index")).to.deep.equal(0)
       expect(dp.getField(1, "index")).to.deep.equal(1)
@@ -115,7 +115,7 @@ describe("data_table module", () => {
     it("should return fields when sorted", () => {
       const source = new ColumnDataSource({data: {index: [0,1,2,10], bar: [3.4, 1.2, 0, -10]}})
       const view = new CDSView({source})
-      const dp = new DataProvider(source, view)
+      const dp = new TableDataProvider(source, view)
 
       const fake_col = {sortAsc: true, sortCol: {field: "bar"}}
       dp.sort([fake_col])
@@ -139,7 +139,7 @@ describe("data_table module", () => {
     it("should get all records", () => {
       const source = new ColumnDataSource({data: {index: [0,1,2,10], bar: [3.4, 1.2, 0, -10]}})
       const view = new CDSView({source})
-      const dp = new DataProvider(source, view)
+      const dp = new TableDataProvider(source, view)
       expect(dp.getRecords()).to.deep.equal(range(0, dp.getLength()).map((i) => dp.getItem(i)))
 
       const fake_col = {sortAsc: true, sortCol: {field: "bar"}}
@@ -150,7 +150,7 @@ describe("data_table module", () => {
     it("should re-order only the index when sorted", () => {
       const source = new ColumnDataSource({data: {index: [0,1,2,10], bar: [3.4, 1.2, 0, -10]}})
       const view = new CDSView({source})
-      const dp = new DataProvider(source, view)
+      const dp = new TableDataProvider(source, view)
       expect(dp.index).to.deep.equal([0,1,2,3])
 
       const fake_col = {sortAsc: true, sortCol: {field: "bar"}}
@@ -162,14 +162,14 @@ describe("data_table module", () => {
     it("should return null metadata", () => {
       const source = new ColumnDataSource({data: {index: [0,1,2,10], bar: [3.4, 1.2, 0, -10]}})
       const view = new CDSView({source})
-      const dp = new DataProvider(source, view)
+      const dp = new TableDataProvider(source, view)
       expect(dp.getItemMetadata(0)).to.be.null
     })
 
     it("should set fields when unsorted", () => {
       const source = new ColumnDataSource({data: {index: [0,1,2,10], bar: [3.4, 1.2, 0, -10]}})
       const view = new CDSView({source})
-      const dp = new DataProvider(source, view)
+      const dp = new TableDataProvider(source, view)
 
       dp.setField(0, "index", 10.1)
       expect(dp.source.data).to.deep.equal({index: [10.1,1,2,10], bar: [3.4, 1.2, 0, -10]})
@@ -181,7 +181,7 @@ describe("data_table module", () => {
     it("should set fields when sorted", () => {
       const source = new ColumnDataSource({data: {index: [0,1,2,10], bar: [3.4, 1.2, 0, -10]}})
       const view = new CDSView({source})
-      const dp = new DataProvider(source, view)
+      const dp = new TableDataProvider(source, view)
       const fake_col = {sortAsc: true, sortCol: {field: "bar"}}
       dp.sort([fake_col])
 
diff --git a/bokehjs/test/models/widgets/tables/index.ts b/bokehjs/test/models/widgets/tables/index.ts
--- a/bokehjs/test/models/widgets/tables/index.ts
+++ b/bokehjs/test/models/widgets/tables/index.ts
@@ -1,2 +1,3 @@
 import "./cell_formatters"
 import "./data_table"
+import "./data_cube"
diff --git a/bokehjs/test/size.ts b/bokehjs/test/size.ts
--- a/bokehjs/test/size.ts
+++ b/bokehjs/test/size.ts
@@ -9,7 +9,7 @@ const LIMITS: {[key: string]: number} = {
   // js
   "js/bokeh.min.js":           750,
   "js/bokeh-widgets.min.js":   100,
-  "js/bokeh-tables.min.js":    230,
+  "js/bokeh-tables.min.js":    270,
   "js/bokeh-api.min.js":        90,
   "js/bokeh-gl.min.js":         70,
   // css
diff --git a/tests/baselines/examples/models/file/data_cube b/tests/baselines/examples/models/file/data_cube
new file mode 100644
--- /dev/null
+++ b/tests/baselines/examples/models/file/data_cube
@@ -0,0 +1 @@
+DataCube bbox=[0, 0, 600, 400]
diff --git a/tests/codebase/test_no_client_server_common.py b/tests/codebase/test_no_client_server_common.py
--- a/tests/codebase/test_no_client_server_common.py
+++ b/tests/codebase/test_no_client_server_common.py
@@ -19,6 +19,7 @@
 
 # Standard library imports
 from subprocess import PIPE, Popen
+from sys import executable
 
 # External imports
 
@@ -47,7 +48,7 @@ def test_no_client_server_common():
 
     '''
     code = "import sys; %s; sys.exit(1 if any(('bokeh.client' in x or 'bokeh.server' in x) for x in sys.modules.keys()) else 0)"
-    proc = Popen(["python", "-c", code % ";".join(BASIC_IMPORTS)],stdout=PIPE)
+    proc = Popen([executable, "-c", code % ";".join(BASIC_IMPORTS)],stdout=PIPE)
     proc.wait()
     if proc.returncode != 0:
         assert False
diff --git a/tests/codebase/test_no_ipython_common.py b/tests/codebase/test_no_ipython_common.py
--- a/tests/codebase/test_no_ipython_common.py
+++ b/tests/codebase/test_no_ipython_common.py
@@ -19,6 +19,7 @@
 
 # Standard library imports
 from subprocess import PIPE, Popen
+from sys import executable
 
 # External imports
 
@@ -46,7 +47,7 @@ def test_no_ipython_common():
 
     '''
     proc = Popen([
-        "python", "-c", "import sys; %s; sys.exit(1 if any('IPython' in x for x in sys.modules.keys()) else 0)" % ";".join(BASIC_IMPORTS)
+        executable, "-c", "import sys; %s; sys.exit(1 if any('IPython' in x for x in sys.modules.keys()) else 0)" % ";".join(BASIC_IMPORTS)
     ],stdout=PIPE)
     proc.communicate()
     proc.wait()
diff --git a/tests/codebase/test_no_tornado_common.py b/tests/codebase/test_no_tornado_common.py
--- a/tests/codebase/test_no_tornado_common.py
+++ b/tests/codebase/test_no_tornado_common.py
@@ -19,6 +19,7 @@
 
 # Standard library imports
 from subprocess import PIPE, Popen
+from sys import executable
 
 # External imports
 
@@ -44,7 +45,7 @@ def test_no_tornado_common():
 
     '''
     proc = Popen([
-        "python", "-c", "import sys; %s; sys.exit(1 if any('tornado' in x for x in sys.modules.keys()) else 0)" % ";".join(BASIC_IMPORTS)
+        executable, "-c", "import sys; %s; sys.exit(1 if any('tornado' in x for x in sys.modules.keys()) else 0)" % ";".join(BASIC_IMPORTS)
     ],stdout=PIPE)
     proc.communicate()
     proc.wait()
diff --git a/tests/codebase/test_python_execution_with_OO.py b/tests/codebase/test_python_execution_with_OO.py
--- a/tests/codebase/test_python_execution_with_OO.py
+++ b/tests/codebase/test_python_execution_with_OO.py
@@ -20,6 +20,7 @@
 # Standard library imports
 import os
 from subprocess import PIPE, Popen
+from sys import executable
 
 # External imports
 
@@ -63,7 +64,7 @@ def test_python_execution_with_OO():
     test_env = os.environ.copy()
     test_env['BOKEH_DOCS_MISSING_API_KEY_OK'] = 'yes'
 
-    proc = Popen(["python", "-OO", "-"], stdout=PIPE, stdin=PIPE, env=test_env)
+    proc = Popen([executable, "-OO", "-"], stdout=PIPE, stdin=PIPE, env=test_env)
     proc.communicate("\n".join(imports).encode("utf-8"))
     proc.wait()