Repository: dbt-labs/dbt-core

Issue:
Postgres/Redshift adapters mask SSL syscall error with "connection already closed"
## Issue

### Issue description
A recent Sinter run of `dbt archive` encountered the dreaded:

```
SSL SYSCALL error: EOF detected
```

Relevant part of the call stack:

```
  File "/usr/local/lib/python3.5/dist-packages/dbt/runner.py", line 89, in call_runner
    result = runner.safe_run(manifest)
  File "/usr/local/lib/python3.5/dist-packages/dbt/node_runners.py", line 121, in safe_run
    self.adapter.release_connection(self.profile, node_name)
  File "/usr/local/lib/python3.5/dist-packages/dbt/adapters/default/impl.py", line 494, in release_connection
    cls.rollback(to_release)
  File "/usr/local/lib/python3.5/dist-packages/dbt/adapters/default/impl.py", line 613, in rollback
    connection.get('handle').rollback()
```

When it tries to rollback the transaction on that connection handle, postgres raises another exception which masks the original one and prints out `connection already closed`.

There was an earlier `Failed to release connection!` error -- I don't know if the two are related.

### System information
dbt version: 0.11.0
platform: Sinter
python version: 3.5

### Steps to reproduce
Difficult to repro since it depends on Postgres server failing in a specific way. However I imagine manually killing a postgres connection handle in the middle of an archive job would create similar behavior.



Discussion:

Resolving patch:
diff --git a/dbt/exceptions.py b/dbt/exceptions.py
--- a/dbt/exceptions.py
+++ b/dbt/exceptions.py
@@ -69,7 +69,8 @@ def __str__(self, prefix="! "):
         if hasattr(self.msg, 'split'):
             split_msg = self.msg.split("\n")
         else:
-            split_msg = basestring(self.msg).split("\n")
+            # can't use basestring here, as on python2 it's an abstract class
+            split_msg = str(self.msg).split("\n")
 
         lines = ["{}{}".format(self.type + ' Error',
                                node_string)] + split_msg
diff --git a/dbt/node_runners.py b/dbt/node_runners.py
--- a/dbt/node_runners.py
+++ b/dbt/node_runners.py
@@ -16,7 +16,10 @@
 import dbt.templates
 import dbt.writer
 
+import six
+import sys
 import time
+import traceback
 
 
 INTERNAL_ERROR_STRING = """This is an error in dbt. Please try again. If \
@@ -73,6 +76,7 @@ def safe_run(self, manifest):
 
         result = RunModelResult(self.node)
         started = time.time()
+        exc_info = (None, None, None)
 
         try:
             # if we fail here, we still have a compiled node to return
@@ -105,6 +109,9 @@ def safe_run(self, manifest):
             result.status = 'ERROR'
 
         except Exception as e:
+            # set this here instead of finally, as python 2/3 exc_info()
+            # behavior with re-raised exceptions are slightly different
+            exc_info = sys.exc_info()
             prefix = "Unhandled error while executing {filepath}".format(
                         filepath=self.node.build_path)
 
@@ -112,16 +119,41 @@ def safe_run(self, manifest):
                          prefix=dbt.ui.printer.red(prefix),
                          error=str(e).strip())
 
-            logger.debug(error)
+            logger.error(error)
             raise e
 
         finally:
-            node_name = self.node.name
-            self.adapter.release_connection(self.config, node_name)
+            exc_str = self._safe_release_connection()
+
+            # if we had an unhandled exception, re-raise it
+            if exc_info and exc_info[1]:
+                six.reraise(*exc_info)
+
+            # if releasing failed and the result doesn't have an error yet, set
+            # an error
+            if exc_str is not None and result.error is None:
+                result.error = exc_str
+                result.status = 'ERROR'
 
         result.execution_time = time.time() - started
         return result
 
+    def _safe_release_connection(self):
+        """Try to release a connection. If an exception is hit, log and return
+        the error string.
+        """
+        node_name = self.node.name
+        try:
+            self.adapter.release_connection(self.config, node_name)
+        except Exception as exc:
+            logger.debug(
+                'Error releasing connection for node {}: {!s}\n{}'
+                .format(node_name, exc, traceback.format_exc())
+            )
+            return dbt.compat.to_string(exc)
+
+        return None
+
     def before_execute(self):
         raise NotImplementedException()
 


Test patch: