internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: google/timesketch

Issue:
timesketch_cli_client.commands.search bug in timestamp handling
In [cli_client/python/timesketch_cli_client/commands/search.py#L169](https://github.com/google/timesketch/blob/f861eba06c5c8b248dee4610ac6b401a93714331/cli_client/python/timesketch_cli_client/commands/search.py#L169) the `time_range` and `time` args are assigned to methods of `timesketch_api_client.search.DateRangeChip` instead of being used as method arguments.

Example:
```py
                range_chip.add_start_time = time_range[0]
                range_chip.add_end_time = time_range[1]
```

Should be:
```py
                range_chip.add_start_time(time_range[0])
                range_chip.add_end_time(time_range[1])
```


Discussion:

Resolving patch:
diff --git a/cli_client/python/timesketch_cli_client/commands/search.py b/cli_client/python/timesketch_cli_client/commands/search.py
--- a/cli_client/python/timesketch_cli_client/commands/search.py
+++ b/cli_client/python/timesketch_cli_client/commands/search.py
@@ -166,8 +166,8 @@ def search_group(
         for time_range in time_ranges:
             try:
                 range_chip = search.DateRangeChip()
-                range_chip.add_start_time = time_range[0]
-                range_chip.add_end_time = time_range[1]
+                range_chip.add_start_time(time_range[0])
+                range_chip.add_end_time(time_range[1])
                 search_obj.add_chip(range_chip)
             except ValueError:
                 click.echo("Error parsing date (make sure it is ISO formatted)")
@@ -179,8 +179,8 @@ def search_group(
         for time in times:
             try:
                 range_chip = search.DateRangeChip()
-                range_chip.add_start_time = time
-                range_chip.add_end_time = time
+                range_chip.add_start_time(time)
+                range_chip.add_end_time(time)
                 search_obj.add_chip(range_chip)
             except ValueError:
                 click.echo("Error parsing date (make sure it is ISO formatted)")


Test patch:
Repository: google/timesketch

Issue:
Performance impacted during data ingestion   
**Describe the bug**
In the OpenSearch datastore helper module in particular import_event function. a call is made to the function responsible for fetching the running OpenSearch version every time an event is added. this dramatically increases the data ingestion time.

https://github.com/google/timesketch/blob/db986692f1f73614290ba9116f0d28d6b1c01062/timesketch/lib/datastores/opensearch.py#L981


**To Reproduce**
Steps to reproduce the behavior:
1. Import data into Timesketch


**Expected behavior**
The version information should be retrieved once when the OpenSearch connection is first initialized. 



Performance impacted during data ingestion   
**Describe the bug**
In the OpenSearch datastore helper module in particular import_event function. a call is made to the function responsible for fetching the running OpenSearch version every time an event is added. this dramatically increases the data ingestion time.

https://github.com/google/timesketch/blob/db986692f1f73614290ba9116f0d28d6b1c01062/timesketch/lib/datastores/opensearch.py#L981


**To Reproduce**
Steps to reproduce the behavior:
1. Import data into Timesketch


**Expected behavior**
The version information should be retrieved once when the OpenSearch connection is first initialized. 





Discussion:

Resolving patch:
diff --git a/timesketch/lib/datastores/opensearch.py b/timesketch/lib/datastores/opensearch.py
--- a/timesketch/lib/datastores/opensearch.py
+++ b/timesketch/lib/datastores/opensearch.py
@@ -134,6 +134,7 @@ def __init__(self, host="127.0.0.1", port=9200):
 
         self.import_counter = Counter()
         self.import_events = []
+        self.version = self.client.info().get("version").get("number")
         self._request_timeout = current_app.config.get(
             "TIMEOUT_FOR_EVENT_IMPORT", self.DEFAULT_EVENT_IMPORT_TIMEOUT
         )
@@ -1105,13 +1106,3 @@ def flush_queued_events(self, retry_count=0):
 
         self.import_events = []
         return return_dict
-
-    @property
-    def version(self):
-        """Get OpenSearch version.
-
-        Returns:
-          Version number as a string.
-        """
-        version_info = self.client.info().get("version")
-        return version_info.get("number")


Test patch:
Repository: google/timesketch

Issue:
New authentication cannot be performed on terminals with no tty
**Describe the bug**
When authenticating for the first time (or after removing a token) the user is presented with a url to visit via stdout, and a token is entered via stdin. In a session with no, or non default tty's, this means the user will not see the url, and may not be able to enter the token.

**To Reproduce**
Steps to reproduce the behavior:
1. Set up an environment with no standard tty. An example is timesketch api user [DFTimewolf](https://github.com/log2timeline/dftimewolf), which has recently implemented a curses UX. 
2. Remove your timesketch token from `~/.timesketch.token`
3. Run the tool

**Expected behavior**
The user should be notified that the session won't support providing a URL to visit, and cannot accept the required token

**Actual behavior**
The user is not aware they need to reauthenticate, as stdout is not available. A user cannot paste the authentication token into the terminal as stdin is not available. Instead, the application blocks on stdin with no indication of the problem.


Discussion:

Resolving patch:
diff --git a/api_client/python/timesketch_api_client/client.py b/api_client/python/timesketch_api_client/client.py
--- a/api_client/python/timesketch_api_client/client.py
+++ b/api_client/python/timesketch_api_client/client.py
@@ -16,6 +16,7 @@
 
 import os
 import logging
+import sys
 
 # pylint: disable=wrong-import-order
 import bs4
@@ -246,6 +247,11 @@ def _create_oauth_session(
         if run_server:
             _ = flow.run_local_server()
         else:
+            if not sys.stdout.isatty() or not sys.stdin.isatty():
+                msg = ('You will be asked to paste a token into this session to'
+                    'authenticate, but the session doesn\'t have a tty')
+                raise RuntimeError(msg)
+
             auth_url, _ = flow.authorization_url(prompt="select_account")
 
             if skip_open:


Test patch:
Repository: google/timesketch

Issue:
gcs-importer fails to read username when importing Plaso db from GCS
**Describe the bug**
When the gcs-importer attempts to import a plaso file from GCS, it fails to read the username from the "requester" field in the .plaso.metadata.json file. This means the import fails.

Jul 14 07:51:58 timesketch-server python3[14864]: 2022-07-14 07:51:58,892 [INFO] Downloaded file from GCS: /tmp/381929d791d04ec0a12f85454cf08196.plaso.metadata.json
Jul 14 07:51:58 timesketch-server python3[14864]: 2022-07-14 07:51:58 INFO     Downloaded file from GCS: /tmp/381929d791d04ec0a12f85454cf08196.plaso.metadata.json
Jul 14 07:51:59 timesketch-server python3[14864]: 2022-07-14 07:51:59,923 [INFO] Downloaded file from GCS: /tmp/381929d791d04ec0a12f85454cf08196.plaso
Jul 14 07:51:59 timesketch-server python3[14864]: 2022-07-14 07:51:59 INFO     Downloaded file from GCS: /tmp/381929d791d04ec0a12f85454cf08196.plaso
Jul 14 07:51:59 timesketch-server python3[14864]: 2022-07-14 07:51:59,926 [ERROR] Missing username
Jul 14 07:51:59 timesketch-server python3[14864]: 2022-07-14 07:51:59 ERROR    Missing username

Since [this commit](https://github.com/google/turbinia/commit/125696c8c6bd784af435316a020a38fd608dd5e5) in Turbinia, the "requester" field is under "globals" in the .plaso.metadata.json file, while Timesketch attempts to [read it directly from the root level](https://github.com/google/timesketch/blob/e33056e17a10e961dcd4b897c2cd66a1fc12ebae/contrib/gcs_importer.py#L171)

**To Reproduce**
Steps to reproduce the behavior:
1. Install latest Turbinia
2. Install latest Timesketch
3. Set up GCS importer
4. Analyze something with Turbinia and have it drop results to GCS
5. Observe GCS importer fail.

**Expected behavior**
The plaso database is correctly imported and indexed. 

**Screenshots**
If applicable, add screenshots to help explain your problem.

**Desktop (please complete the following information):**
 - OS: [e.g. iOS]
 - Browser [e.g. chrome, safari]
 - Version [e.g. 22]

**Additional context**
Add any other context about the problem here.



Discussion:
Note that I'm happy to make a PR for this myself.
Sure feel free to make a PR; happy to review and merge

Thx for spotting this.

Cheers

Resolving patch:
diff --git a/contrib/gcs_importer.py b/contrib/gcs_importer.py
--- a/contrib/gcs_importer.py
+++ b/contrib/gcs_importer.py
@@ -168,7 +168,10 @@ def callback(message):
 
     with open(local_metadata_file, "r") as metadata_file:
         metadata = json.load(metadata_file)
-        username = metadata.get("requester")
+        username = metadata.get("globals", {}).get("requester")
+        if not username:
+            # Backwards compatibility for old Turbinia versions.
+            username = metadata.get("requester")
         sketch_id_from_metadata = metadata.get("sketch_id")
 
     if not username:


Test patch:
Repository: google/timesketch

Issue:
PLASO_UPPER_MEMORY_LIMIT value of 0 is ignored when running psort.py
**Describe the bug**
If PLASO_UPPER_MEMORY_LIMIT is set to 0, Timesketch does not pass this value through to the --process_memory_limit flag when running psort.py even though 0 is a valid process_memory_limit value for psort.py. Setting the process_memory_limit to 0 on psort.py disables the limit, which is useful in Timesketch when troubleshooting memory issues or processing very large Plaso files. 

**To Reproduce**
Steps to reproduce the behavior:
1. Set PLASO_UPPER_MEMORY_LIMIT to 0 in timesketch.conf
2. Upload a really large timeline that will need more than 4 GB of memory to process
3. Get MemoryError from Plaso when you hit the default 4 GB memory limit

**Expected behavior**
Setting PLASO_UPPER_MEMORY_LIMIT should cause --process_memory_limit to be set to 0 when psort.py is run.

**Screenshots**
N/A

**Desktop (please complete the following information):**
N/A

**Additional context**
N/A


Discussion:

Resolving patch:
diff --git a/timesketch/lib/tasks.py b/timesketch/lib/tasks.py
--- a/timesketch/lib/tasks.py
+++ b/timesketch/lib/tasks.py
@@ -643,8 +643,8 @@ def run_plaso(file_path, events, timeline_name, index_name, source_type, timelin
     if opensearch_ssl:
         cmd.extend(["--use_ssl"])
 
-    psort_memory = current_app.config.get("PLASO_UPPER_MEMORY_LIMIT", "")
-    if psort_memory:
+    psort_memory = current_app.config.get("PLASO_UPPER_MEMORY_LIMIT", None)
+    if psort_memory is not None:
         cmd.extend(["--process_memory_limit", str(psort_memory)])
 
     # Run psort.py


Test patch:
Repository: google/timesketch

Issue:
Submission of a CSV file with malformed headers creates infinite loop of HTTP requests
**Bug Description**
The submission of a CSV with missing mandatory headers fals on the server side. However, even if the clients are notified with an error, they will try to retrieve the timeline using the HTTP requests `/api/v1/sketch/sketch_ID` and `/api/v1/sketch/sketch_ID/timeline/timeline_ID`. These requests are sent until the uploaded CSV file is manually deleted. 

**To Reproduce**
Steps to reproduce the behavior:
1. Go to Timesketch (TS) UI and submit a CSV file that is not compliant with TS specifications (e.g., missing headers).
2. Refresh the page and verify on the info tab of the submission that you have received the error: `Missing fields in CSV header: datetime,timestamp_desc`
3. Open the network tab on your window and verify that your Browser is sending HTTP requests to retrieve the non-created-timeline, thus it will send indefinitely HTTP requests. 

**Expected behavior**
When a CSV file is sent to the Server and some constraints are missed (e.g., mandatory headers are not present), then the Server should return an error that prevents the client to retrieve a non-existing resources

**Screenshots**
The image below shows the error message that the clients receive when they submit a CSV with wrong headers.
 
![errorMessageTS](https://user-images.githubusercontent.com/108743205/179936232-3091bc7e-52b0-4132-a499-3890f8f0728d.png)

The image below shows the HTTP requests after that the user  submits the CSV and after he receives the error message. In a correct scenario, i.e., when the CSV is correctly formatted, these HTTP requests are used to check the status of the uploading of the CSV file into Opensearch.

![infiniteHTTPReq](https://user-images.githubusercontent.com/108743205/179937085-97ef7602-dcda-4e3a-958a-334b43484724.png)


**Additional context**
UI_2237 should partially solve this problem since CSV files with missing mandatory headers can not be uploaded using the UI.




Discussion:

Resolving patch:
diff --git a/timesketch/lib/tasks.py b/timesketch/lib/tasks.py
--- a/timesketch/lib/tasks.py
+++ b/timesketch/lib/tasks.py
@@ -181,6 +181,10 @@ def _set_timeline_status(timeline_id, status, error_msg=None):
     # Check if there is at least one data source that hasn't failed.
     multiple_sources = any(not x.error_message for x in timeline.datasources)
 
+    if error_msg and status=="fail":
+        timeline.set_status(status)
+        timeline.searchindex.set_status(status)
+
     if multiple_sources:
         timeline_status = timeline.get_status.status.lower()
         if timeline_status != "process" and status != "fail":


Test patch:
Repository: google/timesketch

Issue:
Duplicate form fields on upload form
**Describe the bug**
The upload form from a new TImeline displays duplicated `Name` fields

I believe the field was updated to only display if no errors are present, however it looks like the old form field was left in too


```vue
    <!-- timesketch/frontend/src/components/Common/UploadForm.vue -->

      <div class="field" v-if="fileName">
        <label class="label">Name</label>
        <div class="control">
          <input v-model="form.name" class="input" type="text" required placeholder="Name your timeline" />
        </div>
      </div>
      <div class="error" v-if="!error">
        <div class="field" v-if="fileName">
          <label class="label">Name</label>
          <div class="control">
            <input v-model="form.name" class="input" type="text" required placeholder="Name your timeline" />
          </div>
        </div>
```

**To Reproduce**
Steps to reproduce the behavior:
1. Create a new Sketch
2. Go to upload a new timeline
4. See error on the upload form

**Expected behavior**
The `Name` field should only display when there are no errors. 

**Screenshots**
![duplicate-upload-fields](https://user-images.githubusercontent.com/9006404/181908283-b2129c28-58c2-4c08-81a6-a23b8d7b237c.png)


**Desktop (please complete the following information):**
 - OS: Linux
 - Browser chrome




Discussion:

Resolving patch:
diff --git a/importer_client/python/tools/timesketch_importer.py b/importer_client/python/tools/timesketch_importer.py
--- a/importer_client/python/tools/timesketch_importer.py
+++ b/importer_client/python/tools/timesketch_importer.py
@@ -75,6 +75,7 @@ def upload_file(
         able to upload the timeline as well as the celery task identification
         for the indexing.
     """
+
     if not my_sketch or not hasattr(my_sketch, "id"):
         return "Sketch needs to be set"
 
@@ -86,6 +87,10 @@ def upload_file(
             ".jsonl (not {0:s})"
         ).format(file_extension.lower())
 
+    if os.path.getsize(file_path) <= 0:
+        return (
+            "File cannot be empty"
+        )
     import_helper = helper.ImportHelper()
     import_helper.add_config_dict(config_dict)
 
diff --git a/timesketch/api/v1/resources/upload.py b/timesketch/api/v1/resources/upload.py
--- a/timesketch/api/v1/resources/upload.py
+++ b/timesketch/api/v1/resources/upload.py
@@ -344,6 +344,11 @@ def _upload_file(self, file_storage, form, sketch, index_name, chunk_index_name=
         file_size = form.get("total_file_size")
         if isinstance(file_size, str) and file_size.isdigit():
             file_size = int(file_size)
+        if file_size <= 0:
+            abort(
+                HTTP_STATUS_CODE_BAD_REQUEST,
+                "Unable to upload file. File is empty"
+            )
         enable_stream = form.get("enable_stream", False)
 
         data_label = form.get("data_label", "")


Test patch:
Repository: google/timesketch

Issue:
Issue indexing plaso files after uploading
**Describe the bug**
When uploading a plaso file to timesketch via timesketch-importer or through the GUI, we receive the error: 

```
usage: psort.py [-h] [--troubles] [-V] [--analysis PLUGIN_LIST]
                [--process_memory_limit SIZE]
                [--temporary_directory DIRECTORY] [--worker_memory_limit SIZE]
                [--worker_timeout MINUTES] [--logfile FILENAME] [-d] [-q] [-u]
                [--status_view TYPE] [--slice DATE_TIME]
                [--slice_size SLICE_SIZE] [--slicer] [--data PATH] [-a]
                [--language LANGUAGE_TAG] [--dynamic_time]
                [--output_time_zone TIME_ZONE] [-o FORMAT] [-w OUTPUT_FILE]
                [--profilers PROFILERS_LIST] [--profiling_directory DIRECTORY]
                [--profiling_sample_rate SAMPLE_RATE]
                [PATH] [FILTER]
psort.py: error: unrecognized arguments: --server opensearch --port 9200 --index_name 5a96c741d01b401abe20514183df1645 --elastic_mappings /etc/timesketch/plaso.mappings --timeline_identifier 10

```
**To Reproduce**
Steps to reproduce the behavior:
1. Create plaso file
2. upload via GUI OR timesketch-importer.py
3. failure happens immediately after upload
4. See error

**Expected behavior**
Once uploaded the plaso file should index properly.

**Screenshots**
![Screen Shot 2022-08-12 at 11 56 28 AM](https://user-images.githubusercontent.com/55293833/184425321-565605e9-e2dc-48ab-b22c-3ebe09ec213d.png)


**Server:**
Distributor ID:	Ubuntu
Description:	Ubuntu 20.04.4 LTS
Release:	20.04
Codename:	focal





Discussion:
Think this is also causing a e2e tests to fail on PRs [example](https://github.com/google/timesketch/runs/7833775426?check_suite_focus=true).

Resolving patch:
diff --git a/timesketch/lib/tasks.py b/timesketch/lib/tasks.py
--- a/timesketch/lib/tasks.py
+++ b/timesketch/lib/tasks.py
@@ -682,7 +682,7 @@ def run_plaso(
     cmd = [
         psort_path,
         "-o",
-        "elastic_ts",
+        "opensearch_ts",
         file_path,
         "--server",
         opensearch_server,
@@ -695,7 +695,7 @@ def run_plaso(
     ]
 
     if mappings_file_path:
-        cmd.extend(["--elastic_mappings", mappings_file_path])
+        cmd.extend(["--opensearch_mappings", mappings_file_path])
 
     if timeline_id:
         cmd.extend(["--timeline_identifier", str(timeline_id)])


Test patch:
diff --git a/end_to_end_tests/test_data/evtx.plaso b/end_to_end_tests/test_data/evtx.plaso
Binary files a/end_to_end_tests/test_data/evtx.plaso and b/end_to_end_tests/test_data/evtx.plaso differ
Repository: google/timesketch

Issue:
Paginate search result



Discussion:

Resolving patch:
diff --git a/timesketch/api/v1/resources/scenarios.py b/timesketch/api/v1/resources/scenarios.py
--- a/timesketch/api/v1/resources/scenarios.py
+++ b/timesketch/api/v1/resources/scenarios.py
@@ -71,6 +71,7 @@ def get(self, sketch_id):
             )
 
         scenarios = Scenario.query.filter_by(sketch=sketch).all()
+
         return self.to_json(scenarios)
 
     @login_required
diff --git a/timesketch/api/v1/resources/sketch.py b/timesketch/api/v1/resources/sketch.py
--- a/timesketch/api/v1/resources/sketch.py
+++ b/timesketch/api/v1/resources/sketch.py
@@ -114,7 +114,7 @@ def get(self):
             views = (
                 View.query.filter_by(user=current_user, name="")
                 .order_by(View.updated_at.desc())
-                .limit(3)
+                .limit(10)
             )
             sketches = [view.sketch for view in views]
             total_items = len(sketches)


Test patch:
Repository: google/timesketch

Issue:
Redirect URL is missing after OIDC authn
**Describe the bug**
Redirect URL is missing after OIDC authn. The URL parameter is lost when the redirect happen. The solution is to store the parameter in the session, and redirect after the OIDC has been verified (state with CSRF token etc).



Discussion:

Resolving patch:
diff --git a/timesketch/views/auth.py b/timesketch/views/auth.py
--- a/timesketch/views/auth.py
+++ b/timesketch/views/auth.py
@@ -82,6 +82,8 @@ def login():
     # Google OpenID Connect authentication.
     if current_app.config.get("GOOGLE_OIDC_ENABLED", False):
         hosted_domain = current_app.config.get("GOOGLE_OIDC_HOSTED_DOMAIN")
+        # Save the next URL parameter in the session for redirect after login.
+        session['next'] = request.args.get("next", "/")
         return redirect(get_oauth2_authorize_url(hosted_domain))
 
     # Google Identity-Aware Proxy authentication (using JSON Web Tokens)
@@ -410,6 +412,6 @@ def google_openid_connect():
 
     # Log the user in and setup the session.
     if current_user.is_authenticated:
-        return redirect(request.args.get("next") or "/")
+        return redirect(session.get("next", "/"))
 
     return abort(HTTP_STATUS_CODE_BAD_REQUEST, "User is not authenticated.")


Test patch:
Repository: google/timesketch

Issue:
Uploads not working with newest Plaso
**Describe the bug**
When trying to upload a plaso file (I used the test plaso file in this case), the upload fails with `psort.py: error: unrecognized arguments: --elastic_user opensearch --elastic_password opensearch`

**To Reproduce**
Steps to reproduce the behavior:
1. Go to an investigation
2. Click on upload timeline
3. Wait for it to finish uploading
4. See error

**Expected behavior**
The file should upload and index correctly

**Desktop (please complete the following information):**
 - OS: Mac
 - Browser Chrome
 - Version 104.0.5112.101

**Additional context**
I believe the fix here is to change the arguments being passed into `psort.py`. The `elastic_user` argument [here](https://github.com/google/timesketch/blob/843e046b57c4bf3f47400c778f8fbddf5a7e8eb5/timesketch/lib/tasks.py#L661) should be `opensearch_user` and the `elastic_password` [here](https://github.com/google/timesketch/blob/843e046b57c4bf3f47400c778f8fbddf5a7e8eb5/timesketch/lib/tasks.py#L665) should be `opensearch_password`



Discussion:

Resolving patch:
diff --git a/timesketch/lib/tasks.py b/timesketch/lib/tasks.py
--- a/timesketch/lib/tasks.py
+++ b/timesketch/lib/tasks.py
@@ -658,11 +658,11 @@ def run_plaso(file_path, events, timeline_name, index_name, source_type, timelin
 
     opensearch_username = current_app.config.get("OPENSEARCH_USER", "")
     if opensearch_username:
-        cmd.extend(["--elastic_user", opensearch_username])
+        cmd.extend(["--opensearch_user", opensearch_username])
 
     opensearch_password = current_app.config.get("OPENSEARCH_PASSWORD", "")
     if opensearch_password:
-        cmd.extend(["--elastic_password", opensearch_password])
+        cmd.extend(["--opensearch_password", opensearch_password])
 
     opensearch_ssl = current_app.config.get("OPENSEARCH_SSL", False)
     if opensearch_ssl:


Test patch: