Repository: google/turbinia

Issue:
Output manager errors with local installation
Right now if the GCS_OUTPUT_PATH is set when using a "local" installation, the server will error out.  This is not a very common configuration (since a local installation requires shared storage anyway), but we should still make sure that at least the server will not error out when this happens, and it would be nice if this would work as an optional hybrid mode.


```
(turbinia-env) turbinia@ubuntu:~$ turbiniactl -C celeryworker
[INFO] Turbinia version: 20181004
[INFO] Running Turbinia Celery Worker.
 
 -------------- celery@ubuntu v4.3.0 (rhubarb)
---- **** ----- 
--- * ***  * -- Linux-4.18.0-15-generic-x86_64-with-Ubuntu-18.04-bionic 2019-04-08 07:49:32
-- * - **** --- 
- ** ---------- [config]
- ** ---------- .> app:         turbinia:0x7fef00ece510
- ** ---------- .> transport:   redis://localhost:6379/0
- ** ---------- .> results:     redis://localhost:6379/0
- *** --- * --- .> concurrency: 1 (solo)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> myinstance       exchange=myinstance(direct) key=myinstance
                

[tasks]
  . task_runner

[2019-04-08 07:49:32,489: INFO/MainProcess] Connected to redis://localhost:6379/0
[2019-04-08 07:49:32,494: INFO/MainProcess] mingle: searching for neighbors
[2019-04-08 07:49:33,508: INFO/MainProcess] mingle: all alone
[2019-04-08 07:49:33,519: INFO/MainProcess] celery@ubuntu ready.
[2019-04-08 07:49:48,802: INFO/MainProcess] Received task: task_runner[3d405653-3190-4b06-97e8-09a5d3d23d63]  
[2019-04-08 07:49:48,810: INFO/MainProcess] Lock 140664476647568 acquired on /var/tmp/turbinia-worker.lock
[INFO] Starting Task PlasoTask d1c7a261007f4fa8add879c6ac4a9892
[ERROR] PlasoTask Task failed with exception: [global name 'storage' is not defined]
[ERROR] Traceback (most recent call last):
  File "/home/turbinia/turbinia/turbinia/workers/__init__.py", line 560, in run_wrapper
    self.result = self.setup(evidence)
  File "/home/turbinia/turbinia/turbinia/workers/__init__.py", line 457, in setup
    self.output_manager.setup(self)
  File "/home/turbinia/turbinia/turbinia/output_manager.py", line 181, in setup
    self._output_writers = self.get_output_writers(task)
  File "/home/turbinia/turbinia/turbinia/output_manager.py", line 73, in get_output_writers
    local_output_dir=local_output_dir)
  File "/home/turbinia/turbinia/turbinia/output_manager.py", line 360, in __init__
    self.client = storage.Client(project=config.TURBINIA_PROJECT)
NameError: global name 'storage' is not defined

[ERROR] No TurbiniaTaskResult object found after task execution.
[ERROR] Task returned type [<type 'NoneType'>] instead of TurbiniaTaskResult.
[2019-04-08 07:49:48,821: INFO/MainProcess] Lock 140664476647568 released on /var/tmp/turbinia-worker.lock
[2019-04-08 07:49:48,828: ERROR/MainProcess] Task task_runner[3d405653-3190-4b06-97e8-09a5d3d23d63] raised unexpected: TurbiniaException(u'Output Manager is not setup yet.',)
Traceback (most recent call last):
  File "/home/turbinia/turbinia-env/local/lib/python2.7/site-packages/celery/app/trace.py", line 385, in trace_task
    R = retval = fun(*args, **kwargs)
  File "/home/turbinia/turbinia-env/local/lib/python2.7/site-packages/celery/app/trace.py", line 648, in __protected_call__
    return self.run(*args, **kwargs)
  File "/home/turbinia/turbinia/turbinia/task_manager.py", line 79, in task_runner
    return obj.run_wrapper(*args, **kwargs)
  File "/home/turbinia/turbinia/turbinia/workers/__init__.py", line 591, in run_wrapper
    self.result = self.validate_result(self.result)
  File "/home/turbinia/turbinia/turbinia/workers/__init__.py", line 522, in validate_result
    result.setup(self)
  File "/home/turbinia/turbinia/turbinia/workers/__init__.py", line 124, in setup
    raise TurbiniaException('Output Manager is not setup yet.')
TurbiniaException: Output Manager is not setup yet.
```


Discussion:

Resolving patch:
diff --git a/turbinia/config/turbinia_config.py b/turbinia/config/turbinia_config.py
--- a/turbinia/config/turbinia_config.py
+++ b/turbinia/config/turbinia_config.py
@@ -98,7 +98,8 @@
 
 # GCS Path to copy worker results and Evidence output to.
 # Otherwise, set this as 'None' if output will be stored in shared storage.
-GCS_OUTPUT_PATH = 'gs://%s/output' % BUCKET_NAME
+# GCS_OUTPUT_PATH = 'gs://%s/output' % BUCKET_NAME
+GCS_OUTPUT_PATH = None
 
 ################################################################################
 #                           Celery / Redis / Kombu
diff --git a/turbinia/output_manager.py b/turbinia/output_manager.py
--- a/turbinia/output_manager.py
+++ b/turbinia/output_manager.py
@@ -27,7 +27,7 @@
 from turbinia import TurbiniaException
 
 config.LoadConfig()
-if config.TASK_MANAGER.lower() == 'psq':
+if config.GCS_OUTPUT_PATH and config.GCS_OUTPUT_PATH.lower() is not 'none':
   from google.cloud import storage
 
 log = logging.getLogger('turbinia')


Test patch:
diff --git a/turbinia/output_manager_test.py b/turbinia/output_manager_test.py
--- a/turbinia/output_manager_test.py
+++ b/turbinia/output_manager_test.py
@@ -59,6 +59,7 @@ def tearDown(self):
 
   def testGetOutputWriters(self):
     """Tests get_output_writers function for valid response."""
+    output_manager.storage = mock.MagicMock()
     writers = output_manager.OutputManager.get_output_writers(self.task)
     self.assertEquals(len(writers), 2)
     for writer in writers:
@@ -66,6 +67,7 @@ def testGetOutputWriters(self):
 
   def testGetLocalOutputDirs(self):
     """Tests get_local_output_dirs function for valid response."""
+    output_manager.storage = mock.MagicMock()
     self.task.output_manager.setup(self.task)
     tmp_dir, local_dir = self.task.output_manager.get_local_output_dirs()