internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: google/turbinia

Issue:
Cron.py unicode decode error
It looks like input evidence caused a unicode error for cron. I think this would only be raised when receiving unreadable input but filing this if we want to catch this exception differently/look into it further

```
 Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/turbinia-20220216-py3.8.egg/turbinia/workers/__init__.py", line 1005, in run_wrapper
    self.result = self.run(evidence, self.result)
  File "/usr/local/lib/python3.8/dist-packages/turbinia-20220216-py3.8.egg/turbinia/workers/cron.py", line 54, in run
    crontab = input_file.read()
  File "/usr/lib/python3.8/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdc in position 0: invalid continuation byte
```


Discussion:

Resolving patch:
diff --git a/turbinia/workers/cron.py b/turbinia/workers/cron.py
--- a/turbinia/workers/cron.py
+++ b/turbinia/workers/cron.py
@@ -51,7 +51,14 @@ def run(self, evidence, result):
 
     # Read the input file
     with open(evidence.local_path, 'r') as input_file:
-      crontab = input_file.read()
+      try:
+        crontab = input_file.read()
+      except UnicodeDecodeError as exception:
+        message = 'Error parsing cron file {0:s}: {1!s}'.format(
+            evidence.local_path, exception)
+        result.log(message)
+        result.close(self, success=False, status=message)
+        return result
 
     (report, priority, summary) = self.analyse_crontab(crontab)
     output_evidence.text_data = report


Test patch:
Repository: google/turbinia

Issue:
AttributeError: 'NoneType' object has no attribute 'id'  in process_tasks function
Hi @aarontp seeing this new bug come in with new release. I'm not exactly sure what it is but seems to be related to the server timeouts we've been seeing and believe it may also cause the server to crash when it is raised. 

```
 Traceback (most recent call last):
  File "/usr/local/bin/turbiniactl", line 33, in <module>
    sys.exit(load_entry_point('turbinia==20220321', 'console_scripts', 'turbiniactl')())
  File "/usr/local/lib/python3.8/dist-packages/turbinia-20220321-py3.8.egg/turbinia/turbiniactl.py", line 996, in main
    process_args(sys.argv[1:])
  File "/usr/local/lib/python3.8/dist-packages/turbinia-20220321-py3.8.egg/turbinia/turbiniactl.py", line 677, in process_args
    server.start()
  File "/usr/local/lib/python3.8/dist-packages/turbinia-20220321-py3.8.egg/turbinia/server.py", line 64, in start
    self.task_manager.run()
  File "/usr/local/lib/python3.8/dist-packages/turbinia-20220321-py3.8.egg/turbinia/task_manager.py", line 590, in run
    for task in self.process_tasks():
  File "/usr/local/lib/python3.8/dist-packages/turbinia-20220321-py3.8.egg/turbinia/task_manager.py", line 807, in process_tasks
    .format(psq_task.id, timeout))
AttributeError: 'NoneType' object has no attribute 'id' 
```


Discussion:

Resolving patch:
diff --git a/turbinia/task_manager.py b/turbinia/task_manager.py
--- a/turbinia/task_manager.py
+++ b/turbinia/task_manager.py
@@ -804,7 +804,7 @@ def process_tasks(self):
         if timeout:
           log.warning(
               'Task {0:s} timed on server out after {0:d} seconds. Auto-closing Task.'
-              .format(psq_task.id, timeout))
+              .format(task.id, timeout))
           task = self.timeout_task(task, timeout)
           completed_tasks.append(task)
 


Test patch:
Repository: google/turbinia

Issue:
Server side task timeout format string error
Seeing this on the Server when Tasks time out on the Server side.

```ValueError
Unknown format code 'd' for object of type 'str'
Traceback (most recent call last): File "/usr/local/bin/turbiniactl", line 33, in <module> sys.exit(load_entry_point('turbinia==20220325', 'console_scripts', 'turbiniactl')()) File "/usr/local/lib/python3.8/dist-packages/turbinia-20220325-py3.8.egg/turbinia/turbiniactl.py", line 996, in main process_args(sys.argv[1:]) File "/usr/local/lib/python3.8/dist-packages/turbinia-20220325-py3.8.egg/turbinia/turbiniactl.py", line 677, in process_args server.start() File "/usr/local/lib/python3.8/dist-packages/turbinia-20220325-py3.8.egg/turbinia/server.py", line 64, in start self.task_manager.run() File "/usr/local/lib/python3.8/dist-packages/turbinia-20220325-py3.8.egg/turbinia/task_manager.py", line 590, in run for task in self.process_tasks(): File "/usr/local/lib/python3.8/dist-packages/turbinia-20220325-py3.8.egg/turbinia/task_manager.py", line 806, in process_tasks 'Task {0:s} timed on server out after {0:d} seconds. Auto-closing Task.' ValueError: Unknown format code 'd' for object of type 'str'```


Discussion:
Following the code this could only happen if `timeout` is returned in below piece of code as undefined which afaik would result in a str (instead of the expected integer). Setting this at the start check_task_timeout function would solve this.

https://github.com/google/turbinia/blob/4de58abc33f68304161803067e17dd13c7f70ab5/turbinia/task_manager.py#L315

Any other ideas?

Resolving patch:
diff --git a/turbinia/task_manager.py b/turbinia/task_manager.py
--- a/turbinia/task_manager.py
+++ b/turbinia/task_manager.py
@@ -803,7 +803,7 @@ def process_tasks(self):
         timeout = self.check_task_timeout(task)
         if timeout:
           log.warning(
-              'Task {0:s} timed on server out after {0:d} seconds. Auto-closing Task.'
+              'Task {0:s} timed on server out after {1:d} seconds. Auto-closing Task.'
               .format(task.id, timeout))
           task = self.timeout_task(task, timeout)
           completed_tasks.append(task)


Test patch:
Repository: google/turbinia

Issue:
state_manager.get_task_data filtering by task_id does not work
the get_task_data method in the state manager takes in a task_id parameter to be able to filter results by task_id, however, the actual field in Redis is called 'id' so the function does not return any results for an existing task_id:

https://github.com/google/turbinia/blob/master/turbinia/state_manager.py#L281

```
root@95153920ab11:/workspaces/turbinia# turbiniactl status -statistics -t 5b5b9dda1085442197da5f4c07c58b25
2022-04-20 15:57:58 [INFO] Turbinia version: 20220326.dev7+ge9f5d58
2022-04-20 15:57:58 [INFO] Disabling non-allowlisted jobs configured to be disabled in the config file: bulkextractorjob, binaryextractorjob, dfdeweyjob, hindsightjob, volatilityjob, dockercontainersenumerationjob, photorecjob
2022-04-20 15:57:58 [INFO] Registering job timeouts.
No tasks found
```

After editing https://github.com/google/turbinia/blob/master/turbinia/state_manager.py#L281 to check for the 'id' field, the output works as intended:
```
root@95153920ab11:/workspaces/turbinia# turbiniactl status -statistics -t 5b5b9dda1085442197da5f4c07c58b25
2022-04-20 15:58:12 [INFO] Turbinia version: 20220326.dev7+ge9f5d58
2022-04-20 15:58:12 [INFO] Disabling non-allowlisted jobs configured to be disabled in the config file: bulkextractorjob, binaryextractorjob, dfdeweyjob, hindsightjob, volatilityjob, dockercontainersenumerationjob, photorecjob
2022-04-20 15:58:12 [INFO] Registering job timeouts.
Execution time statistics for Turbinia:

All Tasks: Count: 1, Min: 0:00:35, Mean: 0:00:35, Max: 0:00:35
Successful Tasks: Count: 1, Min: 0:00:35, Mean: 0:00:35, Max: 0:00:35
Failed Tasks: Count: 0, Min: None, Mean: None, Max: None
Total Request Time: Count: 1, Min: 0:00:35, Mean: 0:00:35, Max: 0:00:35
Task type CronAnalysisTask: Count: 1, Min: 0:00:35, Mean: 0:00:35, Max: 0:00:35
Worker 95153920ab11: Count: 1, Min: 0:00:35, Mean: 0:00:35, Max: 0:00:35
User root: Count: 1, Min: 0:00:35, Mean: 0:00:35, Max: 0:00:35
```


Discussion:

Resolving patch:
diff --git a/turbinia/state_manager.py b/turbinia/state_manager.py
--- a/turbinia/state_manager.py
+++ b/turbinia/state_manager.py
@@ -278,7 +278,7 @@ def get_task_data(
       # Redis only supports strings; we convert to/from datetime here and below
       tasks = [task for task in tasks if task.get('last_update') > start_time]
     if task_id:
-      tasks = [task for task in tasks if task.get('task_id') == task_id]
+      tasks = [task for task in tasks if task.get('id') == task_id]
     if request_id:
       tasks = [task for task in tasks if task.get('request_id') == request_id]
     if group_id:


Test patch:
Repository: google/turbinia

Issue:
FileArtifactExtraction Tasks failing
From a recent run:
```
* FileArtifactExtractionTask: image_export.py failed for artifact TomcatFiles - local_path not provided.
* FileArtifactExtractionTask: image_export.py failed for artifact SshdConfigFile - local_path not provided.
* FileArtifactExtractionTask: image_export.py failed for artifact RedisConfigFile - local_path not provided.
* FileArtifactExtractionTask: image_export.py failed for artifact JupyterConfigFile - local_path not provided.
* FileArtifactExtractionTask: image_export.py failed for artifact ApacheAccessLogs - local_path not provided.
* FileArtifactExtractionTask: image_export.py failed for artifact NginxAccessLogs - local_path not provided.
* FileArtifactExtractionTask: image_export.py failed for artifact GKEDockerContainerLogs - local_path not provided.
* FileArtifactExtractionTask: image_export.py failed for artifact LinuxScheduleFiles - local_path not provided.
```


Discussion:

Resolving patch:
diff --git a/turbinia/workers/artifact.py b/turbinia/workers/artifact.py
--- a/turbinia/workers/artifact.py
+++ b/turbinia/workers/artifact.py
@@ -27,7 +27,7 @@
 class FileArtifactExtractionTask(TurbiniaTask):
   """Task to run image_export (log2timeline)."""
 
-  REQUIRED_STATES = [state.ATTACHED]
+  REQUIRED_STATES = [state.ATTACHED, state.CONTAINER_MOUNTED]
 
   def __init__(self, artifact_name='FileArtifact'):
     super(FileArtifactExtractionTask, self).__init__()


Test patch:
Repository: google/turbinia

Issue:
Turbiniactl output_manager call with wrong number of arguments
turbiniactl.py calls output_manager.setup() using 3 arguments but the function requires 4.

https://github.com/google/turbinia/blob/4373e6ad723597b488dc3a23508537a07fb6825d/turbinia/turbiniactl.py#L920

https://github.com/google/turbinia/blob/4373e6ad723597b488dc3a23508537a07fb6825d/turbinia/output_manager.py#L221


Discussion:

Resolving patch:
diff --git a/turbinia/turbiniactl.py b/turbinia/turbiniactl.py
--- a/turbinia/turbiniactl.py
+++ b/turbinia/turbiniactl.py
@@ -929,9 +929,13 @@ def process_evidence(
 
   if evidence_ and not args.force_evidence:
     if not config.SHARED_FILESYSTEM and evidence_.copyable:
+      # This is created so we can auto-upload files when they are copyable.
       if os.path.exists(evidence_.local_path):
         output_manager = OutputManager()
-        output_manager.setup(evidence_.type, request_id, remote_only=True)
+        # Passing in request_id as the uid because we don't have an
+        # associated Task ID in this case.
+        output_manager.setup(
+            evidence_.type, request_id, request_id, remote_only=True)
         output_manager.save_evidence(evidence_)
       else:
         msg = (


Test patch:
Repository: google/turbinia

Issue:
Missing sys module import in logger.py
Logger module is missing an import statement for 'sys'


Discussion:

Resolving patch:
diff --git a/turbinia/config/logger.py b/turbinia/config/logger.py
--- a/turbinia/config/logger.py
+++ b/turbinia/config/logger.py
@@ -20,6 +20,7 @@
 import warnings
 import logging.handlers
 import os
+import sys
 
 from turbinia import config
 from turbinia import TurbiniaException


Test patch:
Repository: google/turbinia

Issue:
sphinx docs build broken
Getting an error when trying to build the docs:
```
$ sphinx-build -b html -d build/doctrees docs dist/docs
Running Sphinx v4.5.0
WARNING: html_static_path entry '_static' does not exist
building [mo]: targets for 0 po files that are out of date
building [html]: targets for 19 source files that are out of date
updating environment: [new config] 19 added, 0 changed, 0 removed
reading sources... [  5%] developer/contributing                                                                                                                                                                                                                                                                                                                                                                                        
Extension error (sphinx_markdown_tables):
Handler <function process_tables at 0x7fb9b1b0a700> for event 'source-read' threw an exception (exception: __init__() missing 1 required positional argument: 'config')
```

Trying an earlier version of sphinx and an earlier version of the repo does not resolve the issue.  It seems to be something in the sphinx-markdown-tables module, but that doesn't seem to have changed that recently either (more than a month ago: https://pypi.org/project/sphinx-markdown-tables/0.0.15/#history).


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -34,8 +34,7 @@
 # ones.
 extensions = [
     'sphinx.ext.autodoc', 'sphinx.ext.doctest', 'sphinx.ext.coverage',
-    'sphinx.ext.viewcode', 'sphinx.ext.napoleon', 'sphinx_markdown_tables',
-    'recommonmark'
+    'sphinx.ext.viewcode', 'sphinx.ext.napoleon', 'recommonmark'
 ]
 
 # Add any paths that contain templates here, relative to this directory.


Test patch:
Repository: google/turbinia

Issue:
Set a default file size limit for PlasoTask hashers
Currently, all PlasoTask instances will attempt to hash files of any size, potentially very large ones .This could lead to unusually long processing times.

This is a small part of a larger effort to try to optimize how Turbinia configures Plaso tasks to better utilize inherent parallel processing capabilities.


Discussion:

Resolving patch:
diff --git a/turbinia/workers/plaso.py b/turbinia/workers/plaso.py
--- a/turbinia/workers/plaso.py
+++ b/turbinia/workers/plaso.py
@@ -39,6 +39,7 @@ class PlasoTask(TurbiniaTask):
       # the Plaso documentation
       'status_view': 'none',
       'hashers': 'all',
+      'hasher_file_size_limit': '1073741824',
       'partitions': 'all',
       'vss_stores': 'none',
       # artifact_filters and file_filter are mutually exclusive
@@ -66,9 +67,9 @@ def build_plaso_command(self, base_command, conf):
     cmd = [base_command]
     for k, v in conf.items():
       cli_args = [
-          'status_view', 'hashers', 'partitions', 'vss_stores',
-          'custom_artifact_definitions', 'parsers', 'artifact_filters',
-          'file_filter', 'yara_rules'
+          'status_view', 'hashers', 'hasher_file_size_limit', 'partitions',
+          'vss_stores', 'custom_artifact_definitions', 'parsers',
+          'artifact_filters', 'file_filter', 'yara_rules'
       ]
       if (k not in cli_args or not v):
         continue


Test patch:
Repository: google/turbinia

Issue:
plaso VSS option incorrect
https://github.com/log2timeline/plaso/blob/9cc50c972b257d6cbbea38fa8b39f0bf027e0960/plaso/cli/storage_media_tool.py#L581

^ option should be --no_vss in below location
https://github.com/google/turbinia/blob/86158a95a0b134978628c1680d0997667ec7c935/turbinia/workers/plaso.py#L43

Please check how this will work if recipes pass in the --vss_stores option


Discussion:
The current default for unattended mode is to disable VSS processing, so the `--no_vss` option should not be necessary. However I'll change Plaso UX to deprecate `--no_vss` and support `--vss_stores=none` going forward. Also see: https://github.com/log2timeline/plaso/issues/4150
Looked at this issue today and if i understand correctly, using "--vss_stores none" should not be a problem and we just need to remove any references to the now deprecated "--no_vss" option. Is my understanding correct @joachimmetz ?

Resolving patch:
diff --git a/turbinia/workers/binary_extractor.py b/turbinia/workers/binary_extractor.py
--- a/turbinia/workers/binary_extractor.py
+++ b/turbinia/workers/binary_extractor.py
@@ -101,7 +101,7 @@ def run(self, evidence, result):
 
     cmd = [
         'image_export.py', '--partitions', 'all', '--volumes', 'all',
-        '--no_vss', '--unattended', '--logfile', image_export_log
+        '--vss_stores', 'none', '--unattended', '--logfile', image_export_log
     ]
 
     if self.task_config.get('binary_extraction_path'):


Test patch:
Repository: google/turbinia

Issue:
Config module 'imp' deprecation warning
Currently, the config module uses a deprecated library method to load the config file into a module which causes a DeprecationWarning

```============================================= warnings summary =============================================
turbinia/config/__init__.py:19
  /workspaces/turbinia/turbinia/config/__init__.py:19: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp
```


Discussion:

Resolving patch:
diff --git a/turbinia/config/__init__.py b/turbinia/config/__init__.py
--- a/turbinia/config/__init__.py
+++ b/turbinia/config/__init__.py
@@ -16,12 +16,12 @@
 
 from __future__ import unicode_literals
 
-import imp
+import importlib.util
+import importlib.machinery
 import itertools
 import logging
 import os
 import sys
-
 from turbinia import TurbiniaException
 
 DATETIME_FORMAT = '%Y-%m-%dT%H:%M:%S.%fZ'
@@ -156,7 +156,11 @@ def LoadConfig(config_file=None):
   if 'turbinia_config_tmpl' in config_file:
     log.warning('Using fallback source config. {0:s}'.format(CONFIG_MSG))
   try:
-    _config = imp.load_source('config', config_file)
+    config_loader = importlib.machinery.SourceFileLoader('config', config_file)
+    config_spec = importlib.util.spec_from_loader(
+        config_loader.name, config_loader)
+    _config = importlib.util.module_from_spec(config_spec)
+    config_loader.exec_module(_config)
   except IOError as exception:
     message = (
         'Could not load config file {0:s}: {1!s}'.format(


Test patch:
Repository: google/turbinia

Issue:
EwfDisk is not initialized correctly by turbiniactl
### Problem 

EwfDisk is defined as follows:
https://github.com/google/turbinia/blob/1d1c0ed7d46232dff4cb410197033e8165858ddb/turbinia/evidence.py#L1097-L1099

However, turbiniactl uses the following code to do so.
https://github.com/google/turbinia/blob/1d1c0ed7d46232dff4cb410197033e8165858ddb/turbinia/turbiniactl.py#L910-L911

This results in the following error:
```
Evidence validation failed: Required attribute ewf_path for class EwfDisk is not set. Please check original request.
```

### Suggested Solution

Alter the used named parameters in turbiniactl


Discussion:

Resolving patch:
diff --git a/turbinia/evidence.py b/turbinia/evidence.py
--- a/turbinia/evidence.py
+++ b/turbinia/evidence.py
@@ -1091,7 +1091,7 @@ class EwfDisk(Evidence):
     ewf_path (str): Path to mounted EWF image.
     ewf_mount_path (str): Path to EWF mount directory.
   """
-  REQUIRED_ATTRIBUTES = ['source_path', 'ewf_path', 'ewf_mount_path']
+  REQUIRED_ATTRIBUTES = ['source_path']
   POSSIBLE_STATES = [EvidenceState.ATTACHED]
 
   def __init__(
@@ -1106,8 +1106,10 @@ def __init__(
 
   def _preprocess(self, _, required_states):
     if EvidenceState.ATTACHED in required_states or self.has_child_evidence:
-      self.ewf_mount_path = mount_local.PreprocessMountEwfDisk(self.source_path)
-      self.ewf_path = mount_local.GetEwfDiskPath(self.ewf_mount_path)
+      if not self.ewf_path and not self.ewf_mount_path:
+        self.ewf_mount_path = mount_local.PreprocessMountEwfDisk(
+            self.source_path)
+        self.ewf_path = mount_local.GetEwfDiskPath(self.ewf_mount_path)
       self.device_path = self.ewf_path
       self.local_path = self.ewf_path
       self.state[EvidenceState.ATTACHED] = True


Test patch:
Repository: google/turbinia

Issue:
Create documentation for the Turbinia API command line tool
Write documentation for the new Turbinia API command line tool.


Discussion:

Resolving patch:
diff --git a/turbinia/api/cli/turbinia_client/core/groups.py b/turbinia/api/cli/turbinia_client/core/groups.py
--- a/turbinia/api/cli/turbinia_client/core/groups.py
+++ b/turbinia/api/cli/turbinia_client/core/groups.py
@@ -24,12 +24,12 @@ def config_group():
 
 @click.group('status')
 def status_group():
-  """Get Turbinia request/task status."""
+  """Get Turbinia request or task status."""
 
 
 @click.group('result')
 def result_group():
-  """Get Turbinia task or request results."""
+  """Get Turbinia request or task results."""
 
 
 @click.group('jobs')


Test patch:
Repository: google/turbinia

Issue:
log2timeline not scanning volumes
log2timeline (PlasoHasher/Parser task) is not scanning volumes like the image_export (FileExtractor task) job does. Log2timeline is missing the ```--volumes all``` option.

https://github.com/google/turbinia/blob/7dfde64b24f0e13d1da771e60a00d244d7e2571b/turbinia/workers/binary_extractor.py#L103

https://github.com/google/turbinia/blob/749a25a065e89994d8fb324ebc31530c1b5efa57/turbinia/workers/plaso.py#L150


Discussion:

Resolving patch:
diff --git a/turbinia/workers/plaso.py b/turbinia/workers/plaso.py
--- a/turbinia/workers/plaso.py
+++ b/turbinia/workers/plaso.py
@@ -68,7 +68,7 @@ def build_plaso_command(self, base_command, conf):
       cli_args = [
           'status_view', 'hashers', 'hasher_file_size_limit', 'partitions',
           'vss_stores', 'custom_artifact_definitions', 'parsers',
-          'artifact_filters', 'file_filter', 'yara_rules'
+          'artifact_filters', 'file_filter', 'yara_rules', 'volumes'
       ]
       if (k not in cli_args or not v):
         continue
@@ -148,6 +148,7 @@ class PlasoParserTask(PlasoTask):
       'hashers': 'none',
       'hasher_file_size_limit': None,
       'partitions': 'all',
+      'volumes': 'all',
       'vss_stores': 'none',
       # artifact_filters and file_filter are mutually exclusive
       # parameters and Plaso will error out if both parameters are used.
@@ -170,6 +171,7 @@ class PlasoHasherTask(PlasoTask):
       'hashers': 'all',
       'hasher_file_size_limit': '1073741824',
       'partitions': 'all',
+      'volumes': 'all',
       'vss_stores': 'none',
       # artifact_filters and file_filter are mutually exclusive
       # parameters and Plaso will error out if both parameters are used.


Test patch:
Repository: google/turbinia

Issue:
[FR]: Update GoogleCloudDisk evidence name
### What would this feature improve or what problem would it solve?

Logs will be easier to read because they will have clearer and more specific information about what is being processed.  Also the UI, timeline and other places will also be clearer.

### What is the feature you are proposing?

Update the GoogleCloudDisk Evidence type to include data about the evidence in the evidence name (actual disk name, maybe project, etc).  The default currently just shows up as `GoogleCloudDisk` which isn't very descriptive.  The default evidence object  has a `__str__` method that is something like `return f'{self.type:s}:{self.name:s}:{self.source_path!s}'` and we can use something similar for cloud disks as well.  Maybe `type:project:disk_name` or similar.

### What alternatives have you considered?

_No response_


Discussion:

Resolving patch:
diff --git a/turbinia/evidence.py b/turbinia/evidence.py
--- a/turbinia/evidence.py
+++ b/turbinia/evidence.py
@@ -942,6 +942,13 @@ def __init__(
     self.resource_id = self.disk_name
     self.device_path = None
 
+  @property
+  def name(self):
+    if self._name:
+      return self._name
+    else:
+      return ':'.join((self.type, self.project, self.disk_name))
+
   def _preprocess(self, _, required_states):
     # The GoogleCloudDisk should never need to be mounted unless it has child
     # evidence (GoogleCloudDiskRawEmbedded). In all other cases, the


Test patch: