internlm/SWE-Fixer-Train-110K huggingface dataset:
Repository: google/turbinia

Issue:
Turbinia's setup.py seems to be broken with the latest version of pip
Turbinia can't be isntalled with the latest version of `pip`. (20.1)

```
Collecting pip
  Downloading pip-20.1-py2.py3-none-any.whl (1.5 MB)
Installing collected packages: pip
  Attempting uninstall: pip
    Found existing installation: pip 20.0.2
    Uninstalling pip-20.0.2:
      Successfully uninstalled pip-20.0.2
Successfully installed pip-20.1
[snip]
Collecting turbinia==20190819.6
  Downloading turbinia-20190819.6.tar.gz (88 kB)
    ERROR: Command errored out with exit status 1:
     command: /opt/hostedtoolcache/Python/3.6.10/x64/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-jz1lyg2d/turbinia/setup.py'"'"'; __file__='"'"'/tmp/pip-install-jz1lyg2d/turbinia/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-rm1k5ext
         cwd: /tmp/pip-install-jz1lyg2d/turbinia/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-jz1lyg2d/turbinia/setup.py", line 65, in <module>
        'requirements.txt', session=False)
      File "/tmp/pip-install-jz1lyg2d/turbinia/setup.py", line 64, in <listcomp>
        install_requires=[str(req.req) for req in parse_requirements(
    AttributeError: 'ParsedRequirement' object has no attribute 'req'
```

Works fine on pip 19.1.1.


Discussion:

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -25,11 +25,6 @@
 from setuptools import find_packages
 from setuptools import setup
 
-try:  # for pip >= 10
-  from pip._internal.req import parse_requirements
-except ImportError:  # for pip <= 9.0.3
-  from pip.req import parse_requirements
-
 
 # make sure turbinia is in path
 sys.path.insert(0, '.')
@@ -44,6 +39,9 @@
     'large amounts of evidence, and decreasing response time by parallelizing'
     'processing where possible.')
 
+requirements = []
+with open('requirements.txt','r') as f:
+  requirements = f.read().splitlines()
 setup(
     name='turbinia',
     version=turbinia.__version__,
@@ -63,9 +61,7 @@
     include_package_data=True,
     zip_safe=False,
     entry_points={'console_scripts': ['turbiniactl=turbinia.turbiniactl:main']},
-    install_requires=[str(req.req) for req in parse_requirements(
-        'requirements.txt', session=False)
-    ],
+    install_requires=requirements,
     extras_require={
         'dev': ['mock', 'nose', 'yapf', 'celery~=4.1', 'coverage'],
         'local': ['celery~=4.1', 'kombu~=4.1', 'redis~=3.0'],


Test patch:
Repository: google/turbinia

Issue:
turbiniactl status -u <user> should show full history
The default number of days to show history for is 1, so if you do turbiniactl status -u <user> it doesn't show the full history.  Right now if you want to show full history for a user, you need to specify both flags.  We should change the cloud function to not check the days value when checking the user (unless days is also specified).


Discussion:
@alimez FYI, it looks like the change in #458 to increase the day count has a byproduct of tasks not showing up in `turbiniactl status` once they have started processing.  For now I'll revert the change, but could you potentially take a look?  Thanks!

Resolving patch:
diff --git a/turbinia/client.py b/turbinia/client.py
--- a/turbinia/client.py
+++ b/turbinia/client.py
@@ -697,6 +697,8 @@ def format_task_status(
     Returns:
       String of task status
     """
+    if user and days == 0:
+      days = 1000
     task_results = self.get_task_data(
         instance, project, region, days, task_id, request_id, user)
     if not task_results:


Test patch:
Repository: google/turbinia

Issue:
Configuration file not behaving as expected
I was struggling a bit today with having the configuration file mapped to what I needed it to be (for launching dftimewolf with a one-off configuration of Turbinia).

My ~/.turbiniarc is set to what I want, but the config is still picked up from "whatever file it can find" in the directory pointed to by `TURBINIA_CONFIG_PATH` (specified in `ENVCONFIGVAR`)

This happens because when importing `evidence` (e.g. `from turbinia import evidence`), `LoadConfig` is called with no parameters, and thus populates the config with whatever files it can find there. Subsequent calls to `LoadConfig`, even when passing a `config_file` will still return this first configuration because it has already been loaded.


Discussion:
We could consider *always* loading the configuration if ```LoadConfig``` is called with the ```config_file``` parameter set (even if we already have a ```CONFIG``` initialized). This would behave how I would expect it (I think) -> I change the config file on disk, call ```LoadConfig```, it should update my config.
Change in below location to cover a check if the ```config_file``` param is set and force (re)load the configuration instead of returning the global one.
https://github.com/google/turbinia/blob/master/turbinia/config/__init__.py#L112

Resolving patch:
diff --git a/turbinia/config/__init__.py b/turbinia/config/__init__.py
--- a/turbinia/config/__init__.py
+++ b/turbinia/config/__init__.py
@@ -109,7 +109,7 @@ def LoadConfig(config_file=None):
   # pattern on the config class.
   # pylint: disable=global-statement
   global CONFIG
-  if CONFIG:
+  if CONFIG and not config_file:
     log.debug(
         'Returning cached config from {0:s} instead of reloading config'.format(
             CONFIG.configSource))


Test patch:
Repository: google/turbinia

Issue:
Add retries to tox
Tox fails when trying to check links within our docs if the link is temporarily down/unresponsive. Adding retries to sphinx config should take care of that.


Discussion:

Resolving patch:
diff --git a/docs/conf.py b/docs/conf.py
--- a/docs/conf.py
+++ b/docs/conf.py
@@ -76,6 +76,8 @@
     ]
 }
 
+# Adding retries to linkchecks before declaring a link broken
+linkcheck_retries = 3
 
 # Output file base name for HTML help builder.
 htmlhelp_basename = 'turbiniadoc'


Test patch:
Repository: google/turbinia

Issue:
Client uses hard coded tmp folder to compress evidence
https://github.com/google/turbinia/blob/de550193a1247a4ec48b0797452c6f4b71f67f46/turbinia/turbiniactl.py#L579

Before uploading an evidence folder it is compressed into temporary folder. The code contains the hardcoded path '/tmp'.

Let's change this to python temp paths that work on all operating systems.
https://docs.python.org/3.7/library/tempfile.html -> tempfile.gettempdir()


Discussion:
Actually, I think there is already config.TMP_DIR

Resolving patch:
diff --git a/turbinia/turbiniactl.py b/turbinia/turbiniactl.py
--- a/turbinia/turbiniactl.py
+++ b/turbinia/turbiniactl.py
@@ -576,7 +576,8 @@ def main():
       log.info(
           'A Cloud Only Architecture has been detected. '
           'Compressing the directory for GCS upload.')
-      source_path = archive.CompressDirectory(source_path, output_path='/tmp')
+      source_path = archive.CompressDirectory(
+          source_path, output_path=config.TMP_DIR)
       args.name = args.name if args.name else source_path
       evidence_ = evidence.CompressedDirectory(
           name=args.name, source_path=source_path, source=args.source)


Test patch:
Repository: google/turbinia

Issue:
Crash when running locally
```
$ turbiniactl -t SSHDAnalysisTask -R  rawdisk -l dfchecklist.img 
[INFO] Turbinia version: 20190819
[INFO] Creating request 5d50f281e7fc4a24bd88993ad8bb34a9 with evidence dfchecklist.img
[INFO] Run command "turbiniactl status -r 5d50f281e7fc4a24bd88993ad8bb34a9" to see the status of this request and associated tasks
[INFO] Running Task SSHDAnalysisTask locally
Traceback (most recent call last):
  File "/usr/local/google/home/romaing/venvs/turbinia/bin/turbiniactl", line 11, in <module>
    load_entry_point('turbinia==20190819', 'console_scripts', 'turbiniactl')()
  File "/usr/local/google/home/romaing/venvs/turbinia/lib/python3.8/site-packages/turbinia-20190819-py3.8.egg/turbinia/turbiniactl.py", line 813, in main
    result = client.run_local_task(args.task, request)
  File "/usr/local/google/home/romaing/venvs/turbinia/lib/python3.8/site-packages/turbinia-20190819-py3.8.egg/turbinia/client.py", line 1020, in run_local_task
    result = task.run_wrapper(request.evidence[0])
  File "/usr/local/google/home/romaing/venvs/turbinia/lib/python3.8/site-packages/turbinia-20190819-py3.8.egg/turbinia/workers/__init__.py", line 705, in run_wrapper
    evidence = evidence_decode(evidence)
  File "/usr/local/google/home/romaing/venvs/turbinia/lib/python3.8/site-packages/turbinia-20190819-py3.8.egg/turbinia/evidence.py", line 56, in evidence_decode
    raise TurbiniaException(
turbinia.TurbiniaException: Evidence_dict is not a dictionary, type is <class 'turbinia.evidence.RawDisk'>
```


Discussion:

Resolving patch:
diff --git a/turbinia/client.py b/turbinia/client.py
--- a/turbinia/client.py
+++ b/turbinia/client.py
@@ -1017,7 +1017,7 @@ def run_local_task(self, task_name, request):
     if not request.evidence:
       raise TurbiniaException('TurbiniaRequest does not contain evidence.')
     log.info('Running Task {0:s} locally'.format(task_name))
-    result = task.run_wrapper(request.evidence[0])
+    result = task.run_wrapper(request.evidence[0].serialize())
     return result
 
   def send_request(self, request):


Test patch:
Repository: google/turbinia

Issue:
Prometheus http listener local conflict
When running workers and server locally the prometheus service will try to listen on the same port which creates a (crashing) conflict.


Discussion:

Resolving patch:
diff --git a/turbinia/client.py b/turbinia/client.py
--- a/turbinia/client.py
+++ b/turbinia/client.py
@@ -1142,12 +1142,13 @@ def __init__(self, jobs_denylist=None, jobs_allowlist=None):
 
   def start(self):
     """Start Turbinia Server."""
-    if config.PROMETHEUS_PORT and config.PROMETHEUS_ADDR:
-      log.info('Starting Prometheus endpoint.')
-      start_http_server(
-          port=config.PROMETHEUS_PORT, addr=config.PROMETHEUS_ADDR)
-    else:
-      log.debug('Prometheus config not specified, not starting Prometheus.')
+    if config.PROMETHEUS_ENABLED:
+      if config.PROMETHEUS_PORT and config.PROMETHEUS_ADDR:
+        log.info('Starting Prometheus endpoint.')
+        start_http_server(
+            port=config.PROMETHEUS_PORT, addr=config.PROMETHEUS_ADDR)
+      else:
+        log.info('Prometheus enabled but port or address not set!')
     log.info('Running Turbinia Server.')
     self.task_manager.run()
 
@@ -1269,11 +1270,12 @@ def __init__(self, jobs_denylist=None, jobs_allowlist=None):
 
   def start(self):
     """Start Turbinia PSQ Worker."""
-    if config.PROMETHEUS_PORT and config.PROMETHEUS_ADDR:
-      log.info('Starting Prometheus endpoint.')
-      start_http_server(
-          port=config.PROMETHEUS_PORT, addr=config.PROMETHEUS_ADDR)
-    else:
-      log.debug('Prometheus config not specified, not starting Prometheus.')
+    if config.PROMETHEUS_ENABLED:
+      if config.PROMETHEUS_PORT and config.PROMETHEUS_ADDR:
+        log.info('Starting Prometheus endpoint.')
+        start_http_server(
+            port=config.PROMETHEUS_PORT, addr=config.PROMETHEUS_ADDR)
+      else:
+        log.info('Prometheus enabled but port or address not set!')
     log.info('Running Turbinia PSQ Worker.')
     self.worker.listen()
diff --git a/turbinia/config/turbinia_config_tmpl.py b/turbinia/config/turbinia_config_tmpl.py
--- a/turbinia/config/turbinia_config_tmpl.py
+++ b/turbinia/config/turbinia_config_tmpl.py
@@ -237,6 +237,8 @@
 #
 # These options are required for customizing the prometheus configuration
 ###############################################################################
+# This will enable the Prometheus service for the workers and server.
+PROMETHEUS_ENABLED = True
 
 # Prometheus listen address and port
 PROMETHEUS_ADDR = '0.0.0.0'


Test patch:
Repository: google/turbinia

Issue:
Generate version data from tags and commits
Today we have a hard-coded value in `turbinia/__init__.py`, but it would be nice to generate the version number from the current TAG for releases and from the git commit id when there is no TAG (ie. when running from master or a different branch).


Discussion:
FYI @hacktobeer 

Resolving patch:
diff --git a/setup.py b/setup.py
--- a/setup.py
+++ b/setup.py
@@ -44,7 +44,6 @@
   requirements = f.read().splitlines()
 setup(
     name='turbinia',
-    version=turbinia.__version__,
     description='Automation and Scaling of Digital Forensics Tools',
     long_description=turbinia_description,
     license='Apache License, Version 2.0',
@@ -66,5 +65,7 @@
         'dev': ['mock', 'nose', 'yapf', 'celery~=4.1', 'coverage'],
         'local': ['celery~=4.1', 'kombu~=4.1', 'redis~=3.0'],
         'worker': ['docker-explorer>=20191104', 'plaso>=20200430', 'pyhindsight>=20200607']
-    }
+    },
+    use_scm_version=True,
+    setup_requires=['setuptools_scm']
 )
diff --git a/turbinia/__init__.py b/turbinia/__init__.py
--- a/turbinia/__init__.py
+++ b/turbinia/__init__.py
@@ -14,11 +14,15 @@
 # limitations under the License.
 """Main Turbinia application."""
 
-__version__ = '20190819'
-
 import logging
 log = logging.getLogger('turbinia')
 
+from pkg_resources import get_distribution, DistributionNotFound
+try:
+  __version__ = get_distribution(__name__).version
+except DistributionNotFound:
+  __version__ = "unknown"
+
 
 def log_and_report(message, trace):
   """Log an error and if enabled, send to GCP Error Reporting API.


Test patch:
Repository: google/turbinia

Issue:
Add timestamps to log output
Client log output does not contain timestamps, let's include those.

```


```


Discussion:

Resolving patch:
diff --git a/turbinia/client.py b/turbinia/client.py
--- a/turbinia/client.py
+++ b/turbinia/client.py
@@ -448,14 +448,17 @@ def get_task_data(
       except auth.exceptions.RefreshError as exception:
         if credential_error_count == 0:
           log.info(
-              'GCP Credentials need to be refreshed, please refresh in another '
-              'terminal and this process will resume. Error: {0!s}'.format(
-                  exception))
+              'GCP Credentials need to be refreshed by running gcloud auth '
+              'application-default login, please refresh in another terminal '
+              'and run turbiniactl -w status -r {0!s} and this process will '
+              'resume. Error: {1!s}'.format(request_id, exception))
         else:
           log.debug(
-              'GCP Credentials need to be refreshed, please refresh in another '
-              'terminal and this process will resume. Attempt {0:d}. Error: '
-              '{1!s}'.format(credential_error_count + 1, exception))
+              'GCP Credentials need to be refreshed by running gcloud auth '
+              'application-default login, please refresh in another terminal '
+              'and run turbiniactl -w status -r {0!s} and this process will '
+              'resume. Attempt {1:d}. Error: '
+              '{2!s}'.format(request_id, credential_error_count + 1, exception))
         # Note, we are intentially not incrementing the retry_count here because
         # we will retry indefinitely while we wait for the user to reauth.
         credential_error_count += 1
diff --git a/turbinia/config/logger.py b/turbinia/config/logger.py
--- a/turbinia/config/logger.py
+++ b/turbinia/config/logger.py
@@ -69,7 +69,8 @@ def setup(need_file_handler=True, need_stream_handler=True):
     logger.addHandler(file_handler)
 
   console_handler = logging.StreamHandler()
-  formatter = logging.Formatter('[%(levelname)s] %(message)s')
+  formatter = logging.Formatter(
+      '%(asctime)s [%(levelname)s] %(message)s', "%Y-%m-%d %H:%M:%S")
   console_handler.setFormatter(formatter)
   if need_stream_handler:
     logger.addHandler(console_handler)


Test patch:
Repository: google/turbinia

Issue:
Update google-cloud-logging package
We need to use version >= 2 because of dftimewolf dependencies, and we need to use <=2.0.2 for the google cloud error package, but when trying to use that version I get the following:

```
$ turbiniactl -d server
Traceback (most recent call last):
  File "/home/aaronpeterson/.local/share/virtualenvs/turbinia-aeSTftCa/bin/turbiniactl", line 11, in <module>
    load_entry_point('turbinia', 'console_scripts', 'turbiniactl')()
  File "/home/aaronpeterson/src/turbinia/turbinia/turbiniactl.py", line 428, in main
    from turbinia.lib import google_cloud
  File "/home/aaronpeterson/src/turbinia/turbinia/lib/google_cloud.py", line 33, in <module>
    from google.cloud.logging import _helpers
ImportError: cannot import name '_helpers' from 'google.cloud.logging' (/home/aaronpeterson/.local/share/virtualenvs/turbinia-aeSTftCa/lib/python3.8/site-packages/google/cloud/logging/__init__.py)
```


Discussion:

Resolving patch:
diff --git a/turbinia/lib/google_cloud.py b/turbinia/lib/google_cloud.py
--- a/turbinia/lib/google_cloud.py
+++ b/turbinia/lib/google_cloud.py
@@ -30,8 +30,8 @@
 from googleapiclient.errors import HttpError
 
 from turbinia import TurbiniaException
-from google.cloud.logging import _helpers
-from google.cloud.logging.handlers.transports.background_thread import _Worker
+from google.cloud.logging_v2 import _helpers
+from google.cloud.logging_v2.handlers.transports.background_thread import _Worker
 
 logger = logging.getLogger('turbinia')
 


Test patch:
Repository: google/turbinia

Issue:
docker explorer task failing with debug_tasks enabled
In turbinia config ```DEBUG_TASKS = True```

```
2021-02-24 14:32:11 [INFO] Running sudo /usr/local/bin/de.py -r /tmp/turbinia-mounts/turbinia6k7f8uxc/var/lib/docker list all_containers -d
usage: de.py [-h] [-d] [-r DOCKER_DIRECTORY] [-V]
             {download,mount,list,history} ...
de.py: error: unrecognized arguments: -d
```

https://github.com/hacktobeer/turbinia/blob/master/turbinia/workers/docker.py#L71

^ appends '-d' to the end of the de.py command. This is incorrect and will fail the docker explorer command. It should not be appended but come right after 'de.py' -> 'de.py -d'


Discussion:

Resolving patch:
diff --git a/turbinia/workers/docker.py b/turbinia/workers/docker.py
--- a/turbinia/workers/docker.py
+++ b/turbinia/workers/docker.py
@@ -64,11 +64,13 @@ def GetContainers(self, evidence):
     if not de_binary:
       raise TurbiniaException('Cannot find de.py in path')
 
-    docker_explorer_command = [
-        'sudo', de_binary, '-r', docker_dir, 'list', 'all_containers'
-    ]
+    docker_explorer_command = ['sudo', de_binary]
+
     if config.DEBUG_TASKS or evidence.config.get('debug_tasks'):
       docker_explorer_command.append('-d')
+
+    docker_explorer_command.extend(['-r', docker_dir, 'list', 'all_containers'])
+
     log.info('Running {0:s}'.format(' '.join(docker_explorer_command)))
     try:
       json_string = subprocess.check_output(docker_explorer_command).decode(
diff --git a/turbinia/workers/redis.py b/turbinia/workers/redis.py
--- a/turbinia/workers/redis.py
+++ b/turbinia/workers/redis.py
@@ -57,7 +57,7 @@ def run(self, evidence, result):
     result.report_data = report
 
     # Write the report to the output file.
-    with open(output_file_path, 'w') as fh:
+    with open(output_file_path, 'wb') as fh:
       fh.write(output_evidence.text_data.encode('utf-8'))
 
     # Add the resulting evidence to the result object.


Test patch:
Repository: google/turbinia

Issue:
import TurbiniaException to partitions.py
```
Traceback (most recent call last):
  File "PATH/v2/lib/python3.8/site-packages/turbinia/workers/__init__.py", line 916, in run_wrapper
    self.result = self.run(evidence, self.result)
  File "PATH/v2/lib/python3.8/site-packages/turbinia/workers/partitions.py", line 144, in run
    path_specs = partitions.Enumerate(evidence)
  File "/PATH/v2/lib/python3.8/site-packages/turbinia/processors/partitions.py", line 49, in Enumerate
    raise TurbiniaException(
NameError: name 'TurbiniaException' is not defined

2021-03-05 18:45:56 [ERROR] PartitionEnumerationTask Task failed with exception: [name 'TurbiniaException' is not defined]
```


Discussion:

Resolving patch:
diff --git a/turbinia/processors/partitions.py b/turbinia/processors/partitions.py
--- a/turbinia/processors/partitions.py
+++ b/turbinia/processors/partitions.py
@@ -21,6 +21,7 @@
 from dfvfs.lib import errors as dfvfs_errors
 
 from turbinia.lib.dfvfs_classes import UnattendedVolumeScannerMediator
+from turbinia import TurbiniaException
 
 log = logging.getLogger('turbinia')
 


Test patch:
Repository: google/turbinia

Issue:
path_spec.parent is None exception
```
FsstatTask Task failed with exception: ['NoneType' object has no attribute 'type_indicator']
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/turbinia-20210208.4.dev19+g7c02e5e-py3.6.egg/turbinia/workers/__init__.py", line 881, in run_wrapper
    self.evidence_setup(evidence)
  File "/usr/local/lib/python3.6/dist-packages/turbinia-20210208.4.dev19+g7c02e5e-py3.6.egg/turbinia/workers/__init__.py", line 499, in evidence_setup
    evidence.preprocess(self.tmp_dir, required_states=self.REQUIRED_STATES)
  File "/usr/local/lib/python3.6/dist-packages/turbinia-20210208.4.dev19+g7c02e5e-py3.6.egg/turbinia/evidence.py", line 341, in preprocess
    self._preprocess(tmp_dir, required_states)
  File "/usr/local/lib/python3.6/dist-packages/turbinia-20210208.4.dev19+g7c02e5e-py3.6.egg/turbinia/evidence.py", line 575, in _preprocess
    encryption_type = partitions.GetPartitionEncryptionType(path_spec)
  File "/usr/local/lib/python3.6/dist-packages/turbinia-20210208.4.dev19+g7c02e5e-py3.6.egg/turbinia/processors/partitions.py", line 66, in GetPartitionEncryptionType
    if path_spec.parent.type_indicator == dfvfs_definitions.TYPE_INDICATOR_BDE:
AttributeError: 'NoneType' object has no attribute 'type_indicator'

No state_manager initialized, not updating Task info
Trying last ditch attempt to close result
Task Result was auto-closed from task executor on turbinia-worker-6eaf93e7aee7c9cf-0 likely due to previous failures.  Previous status: [FsstatTask Task failed with exception: ['NoneType' object has no attribute 'type_indicator']]
Task Result was auto-closed from task executor on turbinia-worker-6eaf93e7aee7c9cf-0 likely due to previous failures.  Previous status: [FsstatTask Task failed with exception: ['NoneType' object has no attribute 'type_indicator']]
```


Discussion:

Resolving patch:
diff --git a/turbinia/processors/partitions.py b/turbinia/processors/partitions.py
--- a/turbinia/processors/partitions.py
+++ b/turbinia/processors/partitions.py
@@ -64,6 +64,10 @@ def GetPartitionEncryptionType(path_spec):
     String representing the type of encryption, or None.
   """
   encryption_type = None
+
+  if not path_spec.HasParent():
+    return None
+
   if path_spec.parent.type_indicator == dfvfs_definitions.TYPE_INDICATOR_BDE:
     encryption_type = 'BDE'
   return encryption_type


Test patch: