internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: google/timesketch

Issue:
Wrong app context for CSV task
We need to run the CSV importer task in the correct context.


Discussion:

Resolving patch:
diff --git a/timesketch/lib/tasks.py b/timesketch/lib/tasks.py
--- a/timesketch/lib/tasks.py
+++ b/timesketch/lib/tasks.py
@@ -121,9 +121,10 @@ def run_csv(source_file_path, timeline_name, index_name, username=None):
     total_events = es.import_event(flush_interval, index_name, event_type)
 
     # We are done so let's remove the processing status flag
-    search_index = SearchIndex.query.filter_by(index_name=index_name).first()
-    search_index.status.remove(search_index.status[0])
-    db_session.add(search_index)
-    db_session.commit()
+    with celery.app.app_context():
+        search_index = SearchIndex.query.filter_by(index_name=index_name).first()
+        search_index.status.remove(search_index.status[0])
+        db_session.add(search_index)
+        db_session.commit()
 
     return {u'Events processed': total_events}


Test patch:
Repository: google/timesketch

Issue:
Improve error handing in the UI
When querying events, most backend errors result in a super unhelpful black box up top in the UI that merely says "Error." Ideally, the error message box would instead:

- Provide users an intelligent error so they know what's wrong, e.g. Elasticsearch went away vs there's a problem with their search syntax
- Make it really obvious (e.g. louder and more obnoxious than the little black box) when the error is potentially preventing the user from seeing the data they expect to see.


Discussion:

Resolving patch:
diff --git a/timesketch/api/v1/resources.py b/timesketch/api/v1/resources.py
--- a/timesketch/api/v1/resources.py
+++ b/timesketch/api/v1/resources.py
@@ -76,6 +76,20 @@
 from timesketch.models.story import Story
 
 
+def bad_request(message):
+    """Function to set custom error message for HTTP 400 requests.
+
+    Args:
+        message: Message as string to return to the client.
+
+    Returns: Response object (instance of flask.wrappers.Response)
+
+    """
+    response = jsonify({u'message': message})
+    response.status_code = HTTP_STATUS_CODE_BAD_REQUEST
+    return response
+
+
 class ResourceMixin(object):
     """Mixin for API resources."""
     # Schemas for database model resources
@@ -447,7 +461,6 @@ def create_view_from_form(sketch, form):
 
         return view
 
-
     @login_required
     def get(self, sketch_id):
         """Handles GET request to the resource.


Test patch:
Repository: google/timesketch

Issue:
Pylint not present in requirements.txt
Not pinning version of Pylint makes our build a bit non-deterministic. Pylint's behavior can change between versions and break our build.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -24,6 +24,8 @@
 
 from setuptools import find_packages
 from setuptools import setup
+from pip.req import parse_requirements
+from pip.download import PipSession
 
 timesketch_version = u'20170721'
 
@@ -83,9 +85,7 @@ def check_before_upload():
     include_package_data=True,
     zip_safe=False,
     scripts=[u'tsctl'],
-    install_requires=frozenset([
-        u'Flask', u'Flask-Login', u'Flask-script', u'Flask-SQLAlchemy',
-        u'Flask-Bcrypt', u'Flask-RESTful', u'Flask-WTF', u'Flask-Migrate',
-        u'SQLAlchemy', u'celery', u'redis', u'blinker', u'elasticsearch',
-        u'neo4jrestclient', u'python-dateutil'
-    ]))
+    install_requires=[str(req.req) for req in parse_requirements(
+        "requirements.txt", session=PipSession(),
+    )],
+)


Test patch:
Repository: google/timesketch

Issue:
Export Entire Timelines / Entire Results
Export feature maxes out at 500 (select all for a search result with >= 500 entries). Would be cool to allow entire timelines to be downloaded from Timelines view, or all search results from the Explore view.


Discussion:
The cap is actually 10k events, so something is not right. I'll take a look.
it seems the limit is set by the number of events per page.

set by default in the [dsl_query](https://github.com/google/timesketch/blob/master/timesketch/views/sketch.py#L446). we could increase the size if we trigger an import.
Actually, I was wrong. size field is also overloaded in query_filter i.e. `{u'from': 0, u'time_end': None, u'limit': 40, u'exclude': [], u'indices': [u'1234567890abcdef'], u'time_start': None, u'order': u'asc', u'size': u'10'}`


Resolving patch:
diff --git a/timesketch/views/sketch.py b/timesketch/views/sketch.py
--- a/timesketch/views/sketch.py
+++ b/timesketch/views/sketch.py
@@ -451,6 +451,7 @@ def export(sketch_id):
     # Export more than the 500 first results.
     max_events_to_fetch = 10000
     query_filter[u'limit'] = max_events_to_fetch
+    query_filter[u'size'] = max_events_to_fetch
 
     datastore = ElasticsearchDataStore(
         host=current_app.config[u'ELASTIC_HOST'],


Test patch:
Repository: google/timesketch

Issue:
Yeti analyzer crash
From the celery log:

```
[2019-07-17 09:11:37,463: ERROR/ForkPoolWorker-1] Task timesketch.lib.tasks.run_sketch_analyzer[46d24990-12df-4c88-a79b-a3b5f1c04b01] raised unexpected: TypeError("unsupported operand type(s) for +: 'NoneType' and 'unicode'",)
Traceback (most recent call last):
  File "/opt/timesketch/env/local/lib/python2.7/site-packages/celery/app/trace.py", line 374, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/opt/timesketch/env/local/lib/python2.7/site-packages/timesketch/__init__.py", line 181, in __call__
    return TaskBase.__call__(self, *args, **kwargs)
  File "/opt/timesketch/env/local/lib/python2.7/site-packages/celery/app/trace.py", line 629, in __protected_call__
    return self.run(*args, **kwargs)
  File "/opt/timesketch/env/local/lib/python2.7/site-packages/timesketch/lib/tasks.py", line 339, in run_sketch_analyzer
    result = analyzer.run_wrapper()
  File "/opt/timesketch/env/local/lib/python2.7/site-packages/timesketch/lib/analyzers/interface.py", line 37, in wrapper
    func_return = func(self, *args, **kwargs)
  File "/opt/timesketch/env/local/lib/python2.7/site-packages/timesketch/lib/analyzers/interface.py", line 403, in run_wrapper
    result = self.run()
  File "/opt/timesketch/env/local/lib/python2.7/site-packages/timesketch/lib/analyzers/yetiindicators.py", line 92, in run
    self.get_intrusion_sets()
  File "/opt/timesketch/env/local/lib/python2.7/site-packages/timesketch/lib/analyzers/yetiindicators.py", line 75, in get_intrusion_sets
    self.yeti_api_root + '/entities/filter/',
TypeError: unsupported operand type(s) for +: 'NoneType' and 'unicode'
```


Discussion:

Resolving patch:
diff --git a/timesketch/lib/analyzers/yetiindicators.py b/timesketch/lib/analyzers/yetiindicators.py
--- a/timesketch/lib/analyzers/yetiindicators.py
+++ b/timesketch/lib/analyzers/yetiindicators.py
@@ -88,6 +88,8 @@ def run(self):
         Returns:
             String with summary of the analyzer result
         """
+        if not self.yeti_api_root or not self.yeti_api_key:
+            return 'No Yeti configuration settings found, aborting.'
 
         self.get_intrusion_sets()
         actors_found = []


Test patch:
Repository: google/timesketch

Issue:
UI: Display Data Sources per Timeline & Make them Clickable
Details
-
The request consists of 3 parts:

1) For each of the timelines that part of a sketch, it would be great to have a list of all data sources that it includes and for each of them the number of entries. Ideally, those data sources should be converted from technical (e.g. "syslog:cron:task_run") into more human readable format (e.g. "Cron Jobs").
2) Each of these data sources to be clickable. Upon click, it would open the Search with pre-populated filter for that data type. Ideally, I'd like the view to be different and instead of showing default columns, show all of the columns for that data type.
3) Have a tick-box next to each of the data sources so that multiple of them can be selected. The ideal behaviour would be to be able to select multiple using the check-boxes, or click on any particular one. The difference could be that selecting multiple data sources at once would display them in a "summary" view, while clicking on each separately would show a detailed view (i.e. a view with all the columns for that data type).

Mock-up Solution
-
![TS Data Sources (1 of 2)](https://user-images.githubusercontent.com/1036124/82788330-997e2a00-9eab-11ea-8064-0eb24405f128.jpg)
-
![TS Data Sources (2 of 2)](https://user-images.githubusercontent.com/1036124/83209193-0db21980-a19b-11ea-9e68-a278a8329b76.jpg)

Reasoning
-
**Helps:**
- Prioritise
- Break down analysis
- Determine completeness (analysis & collection)

**To consider:**
- Focused collection VS Comprehensive collection


Discussion:
I've got the first version ready, will do the last touches and submit a commit by the end of this week.

Resolving patch:
diff --git a/timesketch/lib/datastores/elastic.py b/timesketch/lib/datastores/elastic.py
--- a/timesketch/lib/datastores/elastic.py
+++ b/timesketch/lib/datastores/elastic.py
@@ -211,6 +211,10 @@ def build_query(self, sketch_id, query_string, query_filter, query_dsl=None,
             }
 
             for chip in query_filter['chips']:
+                # Exclude chips that the user disabled
+                if not chip.get('active', True):
+                    continue
+
                 if chip['type'] == 'label':
                     labels.append(chip['value'])
 


Test patch:
Repository: google/timesketch

Issue:
Check for proper permission on timeline 
In api/v1/resources/timeline.py, remove_timeline_label() checks that the user has permission "delete" on the sketch this timeline is added to:
https://github.com/google/timesketch/blob/b5776d5da1586ea6824e89fd514418a35f4d1908/timesketch/api/v1/resources/timeline.py#L270

This mean a user with "write" pemission on sketch (but not "delete") can add timelines to a sketch  (they need 'write' permissions on the sketch), then add labels to it (currently, no permission check), but not remove the label (which, in turn, might prevent the timeline from being deleted further, if said label is 'noarchive' for example).



Discussion:

Resolving patch:
diff --git a/timesketch/api/v1/resources/timeline.py b/timesketch/api/v1/resources/timeline.py
--- a/timesketch/api/v1/resources/timeline.py
+++ b/timesketch/api/v1/resources/timeline.py
@@ -267,12 +267,6 @@ def post(self, sketch_id, timeline_id):
                         self._add_label(timeline=timeline, label=label))
                 changed = any(changes)
             elif label_action == 'remove':
-                if not sketch.has_permission(
-                        user=current_user, permission='delete'):
-                    abort(
-                        HTTP_STATUS_CODE_FORBIDDEN,
-                        'The user does not have delete permission on sketch.')
-
                 changes = []
                 for label in labels:
                     changes.append(


Test patch:
Repository: google/timesketch

Issue:
Allow other OIDC providers for authentication
**Is your feature request related to a problem? Please describe.**
Currently authentication using OIDC is only available to Google identity federation

**Describe the solution you'd like**
To be able to use other OIDC providers for authentication 




Discussion:

Resolving patch:
diff --git a/timesketch/lib/google_auth.py b/timesketch/lib/google_auth.py
--- a/timesketch/lib/google_auth.py
+++ b/timesketch/lib/google_auth.py
@@ -38,12 +38,10 @@
 
 from timesketch.lib.definitions import HTTP_STATUS_CODE_OK
 
-
 CSRF_KEY = 'google_oauth2_csrf_token'
 AUTH_URI = 'https://accounts.google.com/o/oauth2/v2/auth'
 DISCOVERY_URL = 'https://accounts.google.com/.well-known/openid-configuration'
 
-
 class JwtValidationError(Exception):
     """Raised when a JSON Web Token cannot be validated."""
 
@@ -91,8 +89,10 @@ def _fetch_oauth2_discovery_document():
     Returns:
         HTTP response.
     """
+    discovery_url = current_app.config.get(
+        'GOOGLE_OIDC_DISCOVERY_URL', DISCOVERY_URL)
     try:
-        resp = requests.get(DISCOVERY_URL)
+        resp = requests.get(discovery_url)
     except requests.exceptions.RequestException as e:
         raise DiscoveryDocumentError(
             'Cannot fetch discovery document: {}'.format(e)) from e
@@ -120,6 +120,7 @@ def get_oauth2_authorize_url(hosted_domain=None):
     Returns:
         Authorization URL.
     """
+    auth_uri = current_app.config.get('GOOGLE_OIDC_AUTH_URL', AUTH_URI)
     csrf_token = _generate_random_token()
     nonce = _generate_random_token()
     redirect_uri = url_for(
@@ -146,7 +147,7 @@ def get_oauth2_authorize_url(hosted_domain=None):
         params['hd'] = hosted_domain
 
     urlencoded_params = urlparse.urlencode(params)
-    google_authorization_url = '{}?{}'.format(AUTH_URI, urlencoded_params)
+    google_authorization_url = '{}?{}'.format(auth_uri, urlencoded_params)
     return google_authorization_url
 
 
@@ -199,7 +200,9 @@ def decode_jwt(encoded_jwt, public_key, algorithm, expected_audience):
     Args:
         encoded_jwt: The contents of the X-Goog-IAP-JWT-Assertion header.
         public_key: Key to verify signature of the JWT.
-        algorithm: Algorithm used for the key. E.g. ES256, RS256
+        algorithm: Algorithm used for the key. E.g. ES256, RS256. If the
+            GOOGLE_OIDC_ALGORITHM is set in the config, it will overwrite
+            the algorithm used here.
         expected_audience: Expected audience in the JWT.
 
     Returns:
@@ -208,9 +211,12 @@ def decode_jwt(encoded_jwt, public_key, algorithm, expected_audience):
     Raises:
         JwtValidationError: if the JWT token cannot be decoded.
     """
+    chosen_algorithm = current_app.config.get(
+        'GOOGLE_OIDC_ALGORITHM', algorithm)
     try:
         decoded_jwt = jwt.decode(
-            jwt=encoded_jwt, key=public_key, algorithms=[algorithm],
+            jwt=encoded_jwt, key=public_key,
+            algorithms=[chosen_algorithm],
             audience=expected_audience)
         return decoded_jwt
     except (jwt.exceptions.InvalidTokenError,


Test patch:
Repository: google/timesketch

Issue:
Change default search operator (OR --> AND)
The default search operator for string queries is OR. This is somewhat unintuitive, as the defactor standard for search is to have AND as default operator.


Discussion:

Resolving patch:
diff --git a/timesketch/lib/datastores/elastic.py b/timesketch/lib/datastores/elastic.py
--- a/timesketch/lib/datastores/elastic.py
+++ b/timesketch/lib/datastores/elastic.py
@@ -338,7 +338,10 @@ def build_query(self, sketch_id, query_string, query_filter, query_dsl=None,
 
         if query_string:
             query_dsl['query']['bool']['must'].append(
-                {'query_string': {'query': query_string}})
+                {'query_string': {
+                    'query': query_string,
+                    'default_operator': 'AND'}
+                })
 
         # New UI filters
         if query_filter.get('chips', None):


Test patch:
Repository: google/timesketch

Issue:
tagger analyzer not functiong properly 
**Describe the bug**
After upgrade TimeSketch to version: 20210602 the tagger analyzer is not functioning with custom tags

**To Reproduce**
Steps to reproduce the behavior:
1. Import plaso file with evtx data 
2. Add the following tagging rule to tags.yaml
```yaml
logon_tagger:  
  query_string: 'data_type: "windows:evtx:record" AND source_name: "Microsoft-Windows-Security-Auditing" AND event_identifier: 4688'
  tags: ['logon']
  save_search: true
  search_name: 'logon'
```
3. run tagger analyzer
4. See error

**Expected behavior**
The tagger analyzer to run correctly as in previous versions.

**Desktop (please complete the following information):**
-OS:Ubuntu 20.04.2 LTS
-Browser : Firefox
-Version: 86.0

**Additional context**
The following exception is thrown once the tagger analyzer is ran:
```
Traceback (most recent call last): File "/usr/local/lib/python3.8/dist-packages/timesketch/lib/analyzers/interface.py", line 995, in run_wrapper result = self.run() File "/usr/local/lib/python3.8/dist-packages/timesketch/lib/analyzers/tagger.py", line 48, in run tag_result = self.tagger(name, tag_config) File "/usr/local/lib/python3.8/dist-packages/timesketch/lib/analyzers/tagger.py", line 100, in tagger if expression: UnboundLocalError: local variable 'expression' referenced before assignment
``` 



Discussion:

Resolving patch:
diff --git a/timesketch/lib/analyzers/tagger.py b/timesketch/lib/analyzers/tagger.py
--- a/timesketch/lib/analyzers/tagger.py
+++ b/timesketch/lib/analyzers/tagger.py
@@ -83,6 +83,7 @@ def tagger(self, name, config):
 
         expression_string = config.get('regular_expression', '')
         attributes = None
+        expression = None
         if expression_string:
             expression = utils.compile_regular_expression(
                 expression_string=expression_string,


Test patch:
Repository: google/timesketch

Issue:
Sigma analyzer creates saved search and story even with 0 results
The Sigma analyzer will create a story and saved search even if there are no results from the analyzer.

Fix: Only create saved search and story if there was any events tagged.


Discussion:

Resolving patch:
diff --git a/timesketch/lib/analyzers/sigma_tagger.py b/timesketch/lib/analyzers/sigma_tagger.py
--- a/timesketch/lib/analyzers/sigma_tagger.py
+++ b/timesketch/lib/analyzers/sigma_tagger.py
@@ -110,7 +110,7 @@ def run(self):
         total_tagged_events = sum(tags_applied.values())
         output_strings.append('Applied {0:d} tags'.format(total_tagged_events))
 
-        if sigma_rule_counter > 0:
+        if total_tagged_events > 0:
             self.add_sigma_match_view(sigma_rule_counter)
 
         if len(problem_strings) > 0:


Test patch:
Repository: google/timesketch

Issue:
Unittests fail because Unable to locate package python3-mans-to-es

```E: Unable to locate package python3-mans-to-es
Error: Process completed with exit code 100.

```
https://github.com/google/timesketch/blob/cb62b592ad36e33f54b9a3399b147827ddcfe329/.github/workflows/unit-tests.yml#L59

Need to check if we still need that package.


Discussion:

Resolving patch:
diff --git a/importer_client/python/timesketch_import_client/importer.py b/importer_client/python/timesketch_import_client/importer.py
--- a/importer_client/python/timesketch_import_client/importer.py
+++ b/importer_client/python/timesketch_import_client/importer.py
@@ -629,7 +629,7 @@ def add_file(self, filepath, delimiter=','):
 
         else:
             raise TypeError(
-                'File needs to have a file extension of: .csv, .jsonl, mans or '
+                'File needs to have a file extension of: .csv, .jsonl or '
                 '.plaso')
 
     def add_json(self, json_entry, column_names=None):
diff --git a/timesketch/lib/tasks.py b/timesketch/lib/tasks.py
--- a/timesketch/lib/tasks.py
+++ b/timesketch/lib/tasks.py
@@ -11,7 +11,7 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-"""Celery task for processing Plaso storage or mans files."""
+"""Celery task for processing Plaso storage files."""
 
 from __future__ import unicode_literals
 


Test patch:
Repository: google/timesketch

Issue:
json import timestamp error
**Describe the bug**
Importing json data into into a sketch with a different timestamp produces an error.

**To Reproduce**
Steps to reproduce the behavior:
1. import timestamp in iso format.
2. import timestamp in a different format 
3. See import error.

**Expected behavior**
Data to be imported int the sketch.


**Desktop (please complete the following information):**
-OS:Ubuntu 20.04.2 LTS
-Browser : Firefox
-Version: 86.0

**Additional context**
worker logs
```
[2021-07-03 20:07:10,970] timesketch.elasticsearch/ERROR Unable to upload document: J-37bXoBP3DedzX6aM9Q to index f8c9204e71ad4412ab57d6e6e36aabcd - [400] <mapper_parsing_exception> failed to parse field [timestamp] of type [long] in document with id 'J-37bXoBP3DedzX6aM9Q'. Preview of field's value: '2021-01-19T19:55:38+00:00' [illegal_argument_exception/For input string: "2021-01-19T19:55:38+00:00"]
[2021-07-03 19:43:52,041] timesketch.elasticsearch/ERROR Unable to upload document: Ge3mbXoBP3DedzX6D8ly to index f8c9204e71ad4412ab57d6e6e36aabcd - [400] <mapper_parsing_exception> failed to parse field [timestamp] of type [long] in document with id 'Ge3mbXoBP3DedzX6D8ly'. Preview of field's value: '2021-01-19T19:55:38.846091Z' [illegal_argument_exception/For input string: "2021-01-19T19:55:38.846091Z"]
```



Discussion:
Can you share the json you tried to import? (maybe remove sensitive content) and to be sure, you tried to upload that with the Web UI uploader, right?
Sure the json i'm uploading is attached.
i tried via API on a new sketch and it uploaded some of the events and some through an exception on a field other than timestamp field.and upload it in a sketch with data from before and  none of the events were imported.
Regarding UI upload they all fail 
[test_data.log](https://github.com/google/timesketch/files/6776975/test_data.log)

You must have same format in timestamp field on same index... You use different format time. You should use field "datetime" with format: '%Y-%m-%dT%H:%M:%S.%f" , rather than "timestamp".

Resolving patch:
diff --git a/timesketch/lib/utils.py b/timesketch/lib/utils.py
--- a/timesketch/lib/utils.py
+++ b/timesketch/lib/utils.py
@@ -258,7 +258,8 @@ def read_and_validate_jsonl(file_handle):
                 linedict['datetime'] = dt.isoformat()
             if 'timestamp' not in ld_keys and 'datetime' in ld_keys:
                 try:
-                    linedict['timestamp'] = parser.parse(linedict['datetime'])
+                    linedict['timestamp'] = int(parser.parse(
+                        linedict['datetime']).timestamp() * 1000000)
                 except parser.ParserError:
                     logger.error(
                         'Unable to parse timestamp, skipping line '


Test patch: