Repository: google/timesketch

Issue:
Timesketch Timeout on import jsonl
**Describe the bug**
Error timeout ES when I import jsonl (this happens in a non-constant way).

I run command: "tsctl import --file /data/plaso/vol.jsonl --username admin", and log say:
```
[2021-01-21 09:57:16,813] celery.worker.strategy/INFO Received task: timesketch.lib.tasks.run_csv_jsonl[34b8f46651b94a8d8ad9e29bb06a147c]  
[2021-01-21 09:57:16,819] timesketch.tasks/INFO Index timeline [vol timeline] to index [34b8f46651b94a8d8ad9e29bb06a147c] (source: jsonl)
[2021-01-21 09:58:14,763] timesketch.elasticsearch/ERROR Unable to add events
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 421, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 416, in _make_request
    httplib_response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python3.8/http/client.py", line 268, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
  File "/usr/lib/python3.8/socket.py", line 669, in readinto
    return self._sock.recv_into(b)
socket.timeout: timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/connection/http_urllib3.py", line 241, in perform_request
    response = self.pool.urlopen(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 719, in urlopen
    retries = retries.increment(
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 376, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/lib/python3.8/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 665, in urlopen
    httplib_response = self._make_request(
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 423, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 330, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='elasticsearch', port=9200): Read timed out. (read timeout=10)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/timesketch/lib/datastores/elastic.py", line 830, in flush_queued_events
    results = self.client.bulk(body=self.import_events)
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/client/__init__.py", line 448, in bulk
    return self.transport.perform_request(
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/transport.py", line 351, in perform_request
    status, headers_response, data = connection.perform_request(
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/connection/http_urllib3.py", line 253, in perform_request
    raise ConnectionTimeout("TIMEOUT", str(e), e)
elasticsearch.exceptions.ConnectionTimeout: ConnectionTimeout caused by - ReadTimeoutError(HTTPConnectionPool(host='elasticsearch', port=9200): Read timed out. (read timeout=10))
[2021-01-21 09:58:15,185] celery.app.trace/ERROR Task timesketch.lib.tasks.run_csv_jsonl[34b8f46651b94a8d8ad9e29bb06a147c] raised unexpected: UnboundLocalError("local variable 'results' referenced before assignment")
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/timesketch/app.py", line 198, in __call__
    return TaskBase.__call__(self, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/celery/app/trace.py", line 650, in __protected_call__
    return self.run(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/timesketch/lib/tasks.py", line 563, in run_csv_jsonl
    results = es.flush_queued_events()
  File "/usr/local/lib/python3.8/dist-packages/timesketch/lib/datastores/elastic.py", line 835, in flush_queued_events
    errors_in_upload = results.get('errors', False)
UnboundLocalError: local variable 'results' referenced before assignment
```
**To Reproduce**
Steps to reproduce the behavior:


**Expected behavior**
Read is more long of 10seconds timeout.


**Desktop (please complete the following information):**
 - OS: Linux docker install  (us-docker.pkg.dev/osdfir-registry/timesketch/timesketch   latest              f40bb0265ede        8 days ago          564MB)

**Proposed solution**
I found solution work for me. 
At https://github.com/google/timesketch/blob/master/timesketch/lib/datastores/elastic.py#L830, change by:
```
- results = self.client.bulk(body=self.import_events)
+ results = self.client.bulk(body=self.import_events, request_timeout=30)
```
I think the param timeout should be in constant value... Maybe close of https://github.com/google/timesketch/blob/master/timesketch/lib/datastores/elastic.py#L70 .

Thanks.
Lionel


Discussion:
I see one issue at least here, the fact that you see:
```
UnboundLocalError: local variable 'results' referenced before assignment
```

Adding at least a fix for that.

But do you also get the timeout issue when using the importer client?  The importer client is better supported than `tsctl` for file uplaods.
And also, is this running on the dev docker container, or is this a larger deployment? 

(sounds like the Elastic backend here is struggling and therefore not able to reply within the timeout period).

I also don't see the harm in increasing the request timeout a bit. Maybe instead of having it set to 30, to either have it as a constant that can be more easily changed in code, or to have it configurable in the timesketch.conf file ,so that it can be increased on demand.

Resolving patch:
diff --git a/timesketch/api/v1/resources/explore.py b/timesketch/api/v1/resources/explore.py
--- a/timesketch/api/v1/resources/explore.py
+++ b/timesketch/api/v1/resources/explore.py
@@ -104,13 +104,13 @@ def post(self, sketch_id):
         if '_all' in indices:
             indices = all_indices
 
-        # Remove indices that don't exist from search.
-        indices = utils.validate_indices(indices, self.datastore)
-
         # Make sure that the indices in the filter are part of the sketch.
         # This will also remove any deleted timeline from the search result.
         indices, timeline_ids = get_validated_indices(indices, sketch)
 
+        # Remove indices that don't exist from search.
+        indices = utils.validate_indices(indices, self.datastore)
+
         if not indices:
             abort(
                 HTTP_STATUS_CODE_BAD_REQUEST,
diff --git a/timesketch/lib/datastores/elastic.py b/timesketch/lib/datastores/elastic.py
--- a/timesketch/lib/datastores/elastic.py
+++ b/timesketch/lib/datastores/elastic.py
@@ -76,6 +76,7 @@ class ElasticsearchDataStore(object):
     DEFAULT_STREAM_LIMIT = 5000 # Max events to return when streaming results
 
     DEFAULT_FLUSH_RETRY_LIMIT = 3 # Max retries for flushing the queue.
+    DEFAULT_EVENT_IMPORT_TIMEOUT = '3m' # Timeout value for importing events.
 
     def __init__(self, host='127.0.0.1', port=9200):
         """Create a Elasticsearch client."""
@@ -97,6 +98,8 @@ def __init__(self, host='127.0.0.1', port=9200):
 
         self.import_counter = Counter()
         self.import_events = []
+        self._request_timeout = current_app.config.get(
+            'TIMEOUT_FOR_EVENT_IMPORT', self.DEFAULT_EVENT_IMPORT_TIMEOUT)
 
     @staticmethod
     def _build_labels_query(sketch_id, labels):
@@ -945,7 +948,9 @@ def flush_queued_events(self, retry_count=0):
         }
 
         try:
-            results = self.client.bulk(body=self.import_events)
+            # pylint: disable=unexpected-keyword-arg
+            results = self.client.bulk(
+                body=self.import_events, timeout=self._request_timeout)
         except (ConnectionTimeout, socket.timeout):
             if retry_count >= self.DEFAULT_FLUSH_RETRY_LIMIT:
                 es_logger.error(


Test patch: