Repository: dbt-labs/dbt-core

Issue:
"dbt run" can hang
## Issue
`dbt run` can hang indefinitely

### Issue description
If you have a model that another model depends upon, and the materialization of the first model raises an exception that is neither of the type `dbt.exceptions.CompilationException` nor `dbt.exceptions.RuntimeException`, `dbt run` will deadlock.

This issue is reasonably hard to trigger.

### Results
dbt dumps a big horrible stack trace and then hangs until I press ctrl+c. I expected it to fail gracefully.

### System information
The output of `dbt --version`:
```
installed version: 0.12.2
```

The operating system you're running on:
macOS

The python version you're using (probably the output of `python --version`)
3.6.5

### Steps to reproduce
In as much detail as possible, please provide steps to reproduce the issue. Sample data that triggers the issue, example models, etc are all very helpful here.
1) Create a custom materialization, or just edit the view/table materialization in dbt's internal project. At the first line of the materialization, add `{{ exceptions.raise_dependency_error('x') }}`.
2) Make a project with a couple models. Make the second one depend upon the first, and configure dbt to use the materialization you edited for the first.

`raise_dependency_error` is important as it raises an exception that isn't `dbt.exceptions.CompilationException` or `dbt.exceptions.RuntimeException`, which are explicitly caught. I'm pretty sure it's possible to trigger exceptions besides those two + this one - we should catch this kind of error and turn it into an `error` result.

My minimal repro is below:

```
$ cat macros/oops_materialization.sql
{% materialization oops, default -%}
  {{ exceptions.raise_dependency_error('x') }}
{%- endmaterialization %}
$ cat models/base.sql
{{ config(materialized='oops') }}
select 1 as id
$ cat models/dependent.sql
select * from {{ ref('base') }}
$ cat dbt_project.yml
name: 'debug'
version: '0.1.0'
profile: 'debug'
$ dbt run
Found 2 models, 0 tests, 0 archives, 0 analyses, 96 macros, 2 operations, 0 seed files

14:33:08 | Concurrency: 2 threads (target='default')
14:33:08 |
14:33:08 | 1 of 2 START oops model dbt_postgres.base............................ [RUN]
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/Users/jake/.pyenv/versions/3.6.5/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/Users/jake/.pyenv/versions/3.6.5/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/jake/.pyenv/versions/3.6.5/lib/python3.6/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/Users/jake/src/fishtown/dbt/dbt/runner.py", line 80, in call_runner
    result = runner.safe_run(self.manifest)
  File "/Users/jake/src/fishtown/dbt/dbt/node_runners.py", line 90, in safe_run
    result = self.run(compiled_node, manifest)
  File "/Users/jake/src/fishtown/dbt/dbt/node_runners.py", line 159, in run
    return self.execute(compiled_node, manifest)
  File "/Users/jake/src/fishtown/dbt/dbt/node_runners.py", line 456, in execute
    materialization_macro.generator(context)()
  File "/Users/jake/src/fishtown/dbt/dbt/clients/jinja.py", line 102, in call
    return macro(*args, **kwargs)
  File "/Users/jake/.pyenv/versions/3.6.5/envs/dbt36old/lib/python3.6/site-packages/jinja2/runtime.py", line 575, in __call__
    return self._invoke(arguments, autoescape)
  File "/Users/jake/.pyenv/versions/3.6.5/envs/dbt36old/lib/python3.6/site-packages/jinja2/asyncsupport.py", line 110, in _invoke
    return original_invoke(self, arguments, autoescape)
  File "/Users/jake/.pyenv/versions/3.6.5/envs/dbt36old/lib/python3.6/site-packages/jinja2/runtime.py", line 579, in _invoke
    rv = self._func(*arguments)
  File "dbt-a42f706d92e89d13921e04ab", line 16, in macro
    to_string(environment.call(context, environment.getattr((undefined(name='exceptions') if l_1_exceptions is missing else l_1_exceptions), 'raise_dependency_error'), 'x')),
  File "/Users/jake/.pyenv/versions/3.6.5/envs/dbt36old/lib/python3.6/site-packages/jinja2/sandbox.py", line 427, in call
    return __context.call(__obj, *args, **kwargs)
  File "/Users/jake/.pyenv/versions/3.6.5/envs/dbt36old/lib/python3.6/site-packages/jinja2/runtime.py", line 262, in call
    return __obj(*args, **kwargs)
  File "/Users/jake/src/fishtown/dbt/dbt/exceptions.py", line 218, in raise_dependency_error
    raise DependencyException(msg)
dbt.exceptions.DependencyException: x

^C
Exited because of keyboard interrupt.

Done. PASS=0 ERROR=0 SKIP=0 TOTAL=0
ctrl-c
```


Discussion:

Resolving patch:
diff --git a/core/dbt/compat.py b/core/dbt/compat.py
--- a/core/dbt/compat.py
+++ b/core/dbt/compat.py
@@ -22,9 +22,11 @@
 if WHICH_PYTHON == 2:
     basestring = basestring
     bigint = long
+    import __builtin__ as builtins
 else:
     basestring = str
     bigint = int
+    import builtins
 
 if WHICH_PYTHON == 2:
     from SimpleHTTPServer import SimpleHTTPRequestHandler
diff --git a/core/dbt/exceptions.py b/core/dbt/exceptions.py
--- a/core/dbt/exceptions.py
+++ b/core/dbt/exceptions.py
@@ -1,13 +1,13 @@
-from dbt.compat import basestring
+from dbt.compat import basestring, builtins
 from dbt.logger import GLOBAL_LOGGER as logger
 import re
 
 
-class Exception(BaseException):
+class Exception(builtins.Exception):
     pass
 
 
-class MacroReturn(BaseException):
+class MacroReturn(builtins.BaseException):
     """
     Hack of all hacks
     """
diff --git a/core/dbt/node_runners.py b/core/dbt/node_runners.py
--- a/core/dbt/node_runners.py
+++ b/core/dbt/node_runners.py
@@ -76,7 +76,6 @@ def safe_run(self, manifest):
 
         result = RunModelResult(self.node)
         started = time.time()
-        exc_info = (None, None, None)
 
         try:
             # if we fail here, we still have a compiled node to return
@@ -109,9 +108,6 @@ def safe_run(self, manifest):
             result.status = 'ERROR'
 
         except Exception as e:
-            # set this here instead of finally, as python 2/3 exc_info()
-            # behavior with re-raised exceptions are slightly different
-            exc_info = sys.exc_info()
             prefix = "Unhandled error while executing {filepath}".format(
                         filepath=self.node.build_path)
 
@@ -120,15 +116,12 @@ def safe_run(self, manifest):
                          error=str(e).strip())
 
             logger.error(error)
-            raise e
+            result.error = dbt.compat.to_string(e)
+            result.status = 'ERROR'
 
         finally:
             exc_str = self._safe_release_connection()
 
-            # if we had an unhandled exception, re-raise it
-            if exc_info and exc_info[1]:
-                six.reraise(*exc_info)
-
             # if releasing failed and the result doesn't have an error yet, set
             # an error
             if exc_str is not None and result.error is None:


Test patch: