Repository: google/timesketch

Issue:
UI: Display Data Sources per Timeline & Make them Clickable
Details
-
The request consists of 3 parts:

1) For each of the timelines that part of a sketch, it would be great to have a list of all data sources that it includes and for each of them the number of entries. Ideally, those data sources should be converted from technical (e.g. "syslog:cron:task_run") into more human readable format (e.g. "Cron Jobs").
2) Each of these data sources to be clickable. Upon click, it would open the Search with pre-populated filter for that data type. Ideally, I'd like the view to be different and instead of showing default columns, show all of the columns for that data type.
3) Have a tick-box next to each of the data sources so that multiple of them can be selected. The ideal behaviour would be to be able to select multiple using the check-boxes, or click on any particular one. The difference could be that selecting multiple data sources at once would display them in a "summary" view, while clicking on each separately would show a detailed view (i.e. a view with all the columns for that data type).

Mock-up Solution
-
![TS Data Sources (1 of 2)](https://user-images.githubusercontent.com/1036124/82788330-997e2a00-9eab-11ea-8064-0eb24405f128.jpg)
-
![TS Data Sources (2 of 2)](https://user-images.githubusercontent.com/1036124/83209193-0db21980-a19b-11ea-9e68-a278a8329b76.jpg)

Reasoning
-
**Helps:**
- Prioritise
- Break down analysis
- Determine completeness (analysis & collection)

**To consider:**
- Focused collection VS Comprehensive collection


Discussion:
I've got the first version ready, will do the last touches and submit a commit by the end of this week.

Resolving patch:
diff --git a/timesketch/lib/datastores/elastic.py b/timesketch/lib/datastores/elastic.py
--- a/timesketch/lib/datastores/elastic.py
+++ b/timesketch/lib/datastores/elastic.py
@@ -24,6 +24,7 @@
 
 import six
 
+from dateutil import parser, relativedelta
 from elasticsearch import Elasticsearch
 from elasticsearch.exceptions import ConnectionTimeout
 from elasticsearch.exceptions import NotFoundError
@@ -133,6 +134,50 @@ def _build_events_query(events):
         query_dict = {'query': {'ids': {'values': events_list}}}
         return query_dict
 
+    @staticmethod
+    def _convert_to_time_range(interval):
+        """Convert an interval timestamp into start and end dates.
+
+        Args:
+            interval: Time frame representation
+
+        Returns:
+            Start timestamp in string format.
+            End timestamp in string format.
+        """
+        # return ('2018-12-05T00:00:00', '2018-12-05T23:59:59')
+        TS_FORMAT = '%Y-%m-%dT%H:%M:%S'
+        get_digits = lambda s: int(''.join(filter(str.isdigit, s)))
+        get_alpha = lambda s: ''.join(filter(str.isalpha, s))
+
+        ts_parts = interval.split(' ')
+        # The start date could be 1 or 2 first items
+        start = ' '.join(ts_parts[0:len(ts_parts)-2])
+        minus = get_digits(ts_parts[-2])
+        plus = get_digits(ts_parts[-1])
+        interval = get_alpha(ts_parts[-1])
+
+        start_ts = parser.parse(start)
+
+        rd = relativedelta.relativedelta
+        if interval == 's':
+            start_range = start_ts - rd(seconds=minus)
+            end_range = start_ts + rd(seconds=plus)
+        elif interval == 'm':
+            start_range = start_ts - rd(minutes=minus)
+            end_range = start_ts + rd(minutes=plus)
+        elif interval == 'h':
+            start_range = start_ts - rd(hours=minus)
+            end_range = start_ts + rd(hours=plus)
+        elif interval == 'd':
+            start_range = start_ts - rd(days=minus)
+            end_range = start_ts + rd(days=plus)
+        else:
+            raise RuntimeError('Unable to parse the timestamp: '
+                               + str(interval))
+
+        return start_range.strftime(TS_FORMAT), end_range.strftime(TS_FORMAT)
+
     def build_query(self, sketch_id, query_string, query_filter, query_dsl=None,
                     aggregations=None):
         """Build Elasticsearch DSL query.
@@ -231,10 +276,8 @@ def build_query(self, sketch_id, query_string, query_filter, query_dsl=None,
                     elif chip['operator'] == 'must_not':
                         must_not_filters.append(term_filter)
 
-                elif chip['type'] == 'datetime_range':
-                    start = chip['value'].split(',')[0]
-                    end = chip['value'].split(',')[1]
-                    range_filter = {
+                elif chip['type'].startswith('datetime'):
+                    range_filter = lambda start, end: {
                         'range': {
                             'datetime': {
                                 'gte': start,
@@ -242,7 +285,14 @@ def build_query(self, sketch_id, query_string, query_filter, query_dsl=None,
                             }
                         }
                     }
-                    datetime_ranges['bool']['should'].append(range_filter)
+                    if chip['type'] == 'datetime_range':
+                        start, end = chip['value'].split(',')
+                    elif chip['type'] == 'datetime_interval':
+                        start, end = self._convert_to_time_range(chip['value'])
+                    else:
+                        continue
+                    datetime_ranges['bool']['should'].append(
+                        range_filter(start, end))
 
             label_filter = self._build_labels_query(sketch_id, labels)
             must_filters.append(label_filter)


Test patch: