Repository: google/timesketch

Issue:
UI: Display Data Sources per Timeline & Make them Clickable
Details
-
The request consists of 3 parts:

1) For each of the timelines that part of a sketch, it would be great to have a list of all data sources that it includes and for each of them the number of entries. Ideally, those data sources should be converted from technical (e.g. "syslog:cron:task_run") into more human readable format (e.g. "Cron Jobs").
2) Each of these data sources to be clickable. Upon click, it would open the Search with pre-populated filter for that data type. Ideally, I'd like the view to be different and instead of showing default columns, show all of the columns for that data type.
3) Have a tick-box next to each of the data sources so that multiple of them can be selected. The ideal behaviour would be to be able to select multiple using the check-boxes, or click on any particular one. The difference could be that selecting multiple data sources at once would display them in a "summary" view, while clicking on each separately would show a detailed view (i.e. a view with all the columns for that data type).

Mock-up Solution
-
![TS Data Sources (1 of 2)](https://user-images.githubusercontent.com/1036124/82788330-997e2a00-9eab-11ea-8064-0eb24405f128.jpg)
-
![TS Data Sources (2 of 2)](https://user-images.githubusercontent.com/1036124/83209193-0db21980-a19b-11ea-9e68-a278a8329b76.jpg)

Reasoning
-
**Helps:**
- Prioritise
- Break down analysis
- Determine completeness (analysis & collection)

**To consider:**
- Focused collection VS Comprehensive collection
UI: Display Data Sources per Timeline & Make them Clickable
Details
-
The request consists of 3 parts:

1) For each of the timelines that part of a sketch, it would be great to have a list of all data sources that it includes and for each of them the number of entries. Ideally, those data sources should be converted from technical (e.g. "syslog:cron:task_run") into more human readable format (e.g. "Cron Jobs").
2) Each of these data sources to be clickable. Upon click, it would open the Search with pre-populated filter for that data type. Ideally, I'd like the view to be different and instead of showing default columns, show all of the columns for that data type.
3) Have a tick-box next to each of the data sources so that multiple of them can be selected. The ideal behaviour would be to be able to select multiple using the check-boxes, or click on any particular one. The difference could be that selecting multiple data sources at once would display them in a "summary" view, while clicking on each separately would show a detailed view (i.e. a view with all the columns for that data type).

Mock-up Solution
-
![TS Data Sources (1 of 2)](https://user-images.githubusercontent.com/1036124/82788330-997e2a00-9eab-11ea-8064-0eb24405f128.jpg)
-
![TS Data Sources (2 of 2)](https://user-images.githubusercontent.com/1036124/83209193-0db21980-a19b-11ea-9e68-a278a8329b76.jpg)

Reasoning
-
**Helps:**
- Prioritise
- Break down analysis
- Determine completeness (analysis & collection)

**To consider:**
- Focused collection VS Comprehensive collection


Discussion:
I've got the first version ready, will do the last touches and submit a commit by the end of this week.
I've got the first version ready, will do the last touches and submit a commit by the end of this week.

Resolving patch:
diff --git a/timesketch/api/v1/resources/sketch.py b/timesketch/api/v1/resources/sketch.py
--- a/timesketch/api/v1/resources/sketch.py
+++ b/timesketch/api/v1/resources/sketch.py
@@ -29,6 +29,7 @@
 from sqlalchemy import not_
 
 from timesketch.api.v1 import resources
+from timesketch.api.v1 import utils
 from timesketch.lib import forms
 from timesketch.lib.definitions import HTTP_STATUS_CODE_OK
 from timesketch.lib.definitions import HTTP_STATUS_CODE_CREATED
@@ -205,7 +206,8 @@ def get(self, sketch_id):
                 continue
             stats_per_index[timeline.searchindex.index_name] = {
                 'count': 0,
-                'bytes': 0
+                'bytes': 0,
+                'data_types' : []
             }
 
         try:
@@ -230,6 +232,18 @@ def get(self, sketch_id):
                 'bytes': bytes_on_disk
             }
 
+            # Stats per data type in the index.
+            parameters = {
+                'limit': '100',
+                'field': 'data_type'
+            }
+            result_obj, _ = utils.run_aggregator(
+                sketch.id, aggregator_name='field_bucket',
+                aggregator_parameters=parameters,
+                index=[index_name])
+            stats_per_index[index_name]['data_types'] = result_obj.values
+
+
         if not sketch_indices:
             mappings_settings = {}
         else:
diff --git a/timesketch/api/v1/utils.py b/timesketch/api/v1/utils.py
--- a/timesketch/api/v1/utils.py
+++ b/timesketch/api/v1/utils.py
@@ -41,7 +41,8 @@ def bad_request(message):
     return response
 
 
-def run_aggregator(sketch_id, aggregator_name, aggregator_parameters=None):
+def run_aggregator(sketch_id, aggregator_name, aggregator_parameters=None,
+                   index=None):
     """Run an aggregator and return back results.
 
     Args:
@@ -49,6 +50,7 @@ def run_aggregator(sketch_id, aggregator_name, aggregator_parameters=None):
         aggregator_name (str): the name of the aggregator class to run.
         aggregator_parameters (dict): dict containing the parameters used
             for running the aggregator.
+        index (list): the list of elasticsearch index names to use.
 
     Returns:
         Tuple[Object, Dict]: a tuple containing the aggregator result object
@@ -62,7 +64,7 @@ def run_aggregator(sketch_id, aggregator_name, aggregator_parameters=None):
     if not aggregator_parameters:
         aggregator_parameters = {}
 
-    aggregator = agg_class(sketch_id=sketch_id)
+    aggregator = agg_class(sketch_id=sketch_id, index=index)
 
     chart_type = aggregator_parameters.pop('supported_charts', None)
     chart_color = aggregator_parameters.pop('chart_color', '')


Test patch: