Repository: dbt-labs/dbt-core

Issue:
[CT-2142] [Regression] dbt 1.5.0 return invalid exit codes.
### Is this a regression in a recent version of dbt-core?

- [X] I believe this is a regression in dbt-core functionality
- [X] I have searched the existing issues, and I could not find an existing issue for this regression

### Current Behavior

Errored models and failed tests return 0.

### Expected/Previous Behavior

They should return 1.

### Steps To Reproduce

Run a model with a syntax error on dbt 1.5.0+.

### Relevant log output

```
$ dbt run -s error_model
10:27:28  Running with dbt=1.4.1
10:27:29  Found 39 models, 59 tests, 1 snapshot, 0 analyses, 950 macros, 2 operations, 14 seed files, 4 sources, 2 exposures, 0 metrics
10:27:29
10:27:29
10:27:29  Running 1 on-run-start hook
10:27:29  1 of 1 START hook: elementary.on-run-start.0 ................................... [RUN]
10:27:29  1 of 1 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.00s]
10:27:29
10:27:29  Concurrency: 255 threads (target='postgres')
10:27:29
10:27:29  1 of 1 START sql view model elon_test.error_model .............................. [RUN]
10:27:29  1 of 1 ERROR creating sql view model elon_test.error_model ..................... [ERROR in 0.04s]
10:27:29
10:27:29  Running 1 on-run-end hook
10:27:29  1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN]
10:27:29  1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.00s]
10:27:29
10:27:29
10:27:29  Finished running 1 view model, 2 hooks in 0 hours 0 minutes and 0.66 seconds (0.66s).
10:27:29
10:27:29  Completed with 1 error and 0 warnings:
10:27:29
10:27:29  Database Error in model error_model (models/error_model.sql)
10:27:29    syntax error at or near "as"
10:27:29    LINE 4:     select 'a's as string
10:27:29                            ^
10:27:29    compiled Code at ./target/run/elementary_integration_tests/models/error_model.sql
10:27:29
10:27:29  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
$ echo $?
1
$ dbt run -s error_model
10:28:26  Running with dbt=1.5.0-b1
10:28:27  Unable to do partial parsing because of a version mismatch
10:28:29  Found 39 models, 59 tests, 1 snapshot, 0 analyses, 958 macros, 2 operations, 14 seed files, 4 sources, 2 exposures, 0 metrics, 0 groups
10:28:29
10:28:29
10:28:29  Running 1 on-run-start hook
10:28:29  1 of 1 START hook: elementary.on-run-start.0 ................................... [RUN]
10:28:29  1 of 1 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.00s]
10:28:29
10:28:29  Concurrency: 255 threads (target='postgres')
10:28:29
10:28:29  1 of 1 START sql view model elon_test.error_model .............................. [RUN]
10:28:29  1 of 1 ERROR creating sql view model elon_test.error_model ..................... [ERROR in 0.04s]
10:28:29
10:28:29  Running 1 on-run-end hook
10:28:30  1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN]
10:28:30  1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.00s]
10:28:30
10:28:30
10:28:30  Finished running 1 view model, 2 hooks in 0 hours 0 minutes and 0.70 seconds (0.70s).
10:28:30
10:28:30  Completed with 1 error and 0 warnings:
10:28:30
10:28:30  Database Error in model error_model (models/error_model.sql)
10:28:30    syntax error at or near "as"
10:28:30    LINE 4:     select 'a's as string
10:28:30                            ^
10:28:30    compiled Code at ./target/run/elementary_integration_tests/models/error_model.sql
10:28:30
10:28:30  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1
$ echo $?
0
```


### Environment

```markdown
- OS: macOS
- Python: 3.9.6
- dbt (working version): 1.4.1
- dbt (regression version): 1.5.0-b1
```


### Which database adapter are you using with dbt?

postgres

### Additional Context

_No response_


Discussion:
@elongl Thanks for the early beta testing, and for the quick report! I can reproduce this:

```
$ dbt run
10:49:42  Running with dbt=1.5.0-a1
10:49:42  Found 1 model, 0 tests, 0 snapshots, 0 analyses, 410 macros, 0 operations, 1 seed file, 0 sources, 1 exposure, 0 metrics, 0 groups
10:49:42
10:49:42  Concurrency: 5 threads (target='dev')
10:49:42
10:49:42  1 of 1 START sql view model dbt_jcohen.my_model ................................ [RUN]
10:49:42  1 of 1 ERROR creating sql view model dbt_jcohen.my_model ....................... [ERROR in 0.07s]
10:49:42
10:49:42  Finished running 1 view model in 0 hours 0 minutes and 0.36 seconds (0.36s).
10:49:42
10:49:42  Completed with 1 error and 0 warnings:
10:49:42
10:49:42  Database Error in model my_model (models/my_model.sql)
10:49:42    syntax error at or near ";"
10:49:42    LINE 4:     select asldifjas;dl
10:49:42                                ^
10:49:42    compiled Code at target/run/test/models/my_model.sql
10:49:42
10:49:42  Done. PASS=0 WARN=0 ERROR=1 SKIP=0 TOTAL=1

$ echo $?
0
```

I've queued this up with the other bugs that we've been uncovering after the switchover to new CLI / API.
@jtcohen6 Thanks a lot for addressing.
When you're saying new API, you're only referring to the CLI, right?
As in, there's not going to be a stable Python API soon.
@elongl In v1.5, we are planning to release an initial stable Python API, for a subset of `dbt-core` functionality — programmatic invocations. There's a _lot_ this won't include, but it's a first important step on the way there!

Check out #6356 for some more background & links to relevant issues
Documented behavior: https://docs.getdbt.com/reference/exit-codes

Previous logic for this:
https://github.com/dbt-labs/dbt-core/blob/0c067350da663f87197f179d22462e921a544c64/core/dbt/main.py#L128-L157

Potential way of handling this in `click`: https://click.palletsprojects.com/en/8.1.x/api/#click.MultiCommand.result_callback

This resolves the issue for me locally, though it doesn't account for "unhandled" errors (= exit code 2):
```python
def cli(ctx, **kwargs):
    """An ELT tool for managing your SQL transformations and data models.
    For more documentation on these commands, visit: docs.getdbt.com
    """

@cli.result_callback()
def process_result(command_result: Tuple[Any, bool], **kwargs):
    results, success = command_result
    if success:
        exit_code = ExitCodes.Success.value    # 0
    else:
        exit_code = ExitCodes.ModelError.value # 1

    ctx = click.get_current_context()
    ctx.exit(code=exit_code)
    
    # not accounted for here:
    # ExitCodes.UnhandledError.value (=2) if KeyboardInterrupt/click.Abort or BaseException
```
Thanks for reporting the issue!
I tried dbt 1.5 today and I'm reverting because the our CI validation is using exit code to figure out if it's successful.

I use `exit_code != 0` condition especially for exit code 1 but abort could happen (for instance, in a graceful shutdown of the container running it) so supporting that is clearly still relevant in my opinion.

Resolving patch:
diff --git a/core/dbt/cli/main.py b/core/dbt/cli/main.py
--- a/core/dbt/cli/main.py
+++ b/core/dbt/cli/main.py
@@ -56,6 +56,10 @@ def invoke(self, args: List[str]) -> Tuple[Optional[List], bool]:
                 "callbacks": self.callbacks,
             }
             return cli.invoke(dbt_ctx)
+        except requires.HandledExit as e:
+            return (e.result, e.success)
+        except requires.UnhandledExit as e:
+            raise e.exception
         except click.exceptions.Exit as e:
             # 0 exit code, expected for --version early exit
             if str(e) == "0":
@@ -138,6 +142,7 @@ def cli(ctx, **kwargs):
 @requires.project
 @requires.runtime_config
 @requires.manifest
+@requires.postflight
 def build(ctx, **kwargs):
     """Run all Seeds, Models, Snapshots, and tests in DAG order"""
     task = BuildTask(
@@ -162,6 +167,7 @@ def build(ctx, **kwargs):
 @requires.preflight
 @requires.unset_profile
 @requires.project
+@requires.postflight
 def clean(ctx, **kwargs):
     """Delete all folders in the clean-targets list (usually the dbt_packages and target directories.)"""
     task = CleanTask(ctx.obj["flags"], ctx.obj["project"])
@@ -204,6 +210,7 @@ def docs(ctx, **kwargs):
 @requires.project
 @requires.runtime_config
 @requires.manifest(write=False)
+@requires.postflight
 def docs_generate(ctx, **kwargs):
     """Generate the documentation website for your project"""
     task = GenerateTask(
@@ -232,6 +239,7 @@ def docs_generate(ctx, **kwargs):
 @requires.project
 @requires.runtime_config
 @requires.manifest
+@requires.postflight
 def docs_serve(ctx, **kwargs):
     """Serve the documentation website for your project"""
     task = ServeTask(
@@ -275,6 +283,7 @@ def docs_serve(ctx, **kwargs):
 @requires.project
 @requires.runtime_config
 @requires.manifest
+@requires.postflight
 def compile(ctx, **kwargs):
     """Generates executable SQL from source, model, test, and analysis files. Compiled SQL files are written to the
     target/ directory."""
@@ -300,6 +309,7 @@ def compile(ctx, **kwargs):
 @p.vars
 @p.version_check
 @requires.preflight
+@requires.postflight
 def debug(ctx, **kwargs):
     """Show some helpful information about dbt for debugging. Not to be confused with the --debug option which increases verbosity."""
     task = DebugTask(
@@ -323,6 +333,7 @@ def debug(ctx, **kwargs):
 @requires.preflight
 @requires.unset_profile
 @requires.project
+@requires.postflight
 def deps(ctx, **kwargs):
     """Pull the most recent version of the dependencies listed in packages.yml"""
     task = DepsTask(ctx.obj["flags"], ctx.obj["project"])
@@ -343,6 +354,7 @@ def deps(ctx, **kwargs):
 @p.target
 @p.vars
 @requires.preflight
+@requires.postflight
 def init(ctx, **kwargs):
     """Initialize a new dbt project."""
     task = InitTask(ctx.obj["flags"], None)
@@ -375,6 +387,7 @@ def init(ctx, **kwargs):
 @requires.project
 @requires.runtime_config
 @requires.manifest
+@requires.postflight
 def list(ctx, **kwargs):
     """List the resources in your project"""
     task = ListTask(
@@ -412,6 +425,7 @@ def list(ctx, **kwargs):
 @requires.project
 @requires.runtime_config
 @requires.manifest(write_perf_info=True)
+@requires.postflight
 def parse(ctx, **kwargs):
     """Parses the project and provides information on performance"""
     # manifest generation and writing happens in @requires.manifest
@@ -445,6 +459,7 @@ def parse(ctx, **kwargs):
 @requires.project
 @requires.runtime_config
 @requires.manifest
+@requires.postflight
 def run(ctx, **kwargs):
     """Compile SQL and execute against the current target database."""
     task = RunTask(
@@ -473,6 +488,7 @@ def run(ctx, **kwargs):
 @requires.project
 @requires.runtime_config
 @requires.manifest
+@requires.postflight
 def run_operation(ctx, **kwargs):
     """Run the named macro with any supplied arguments."""
     task = RunOperationTask(
@@ -509,6 +525,7 @@ def run_operation(ctx, **kwargs):
 @requires.project
 @requires.runtime_config
 @requires.manifest
+@requires.postflight
 def seed(ctx, **kwargs):
     """Load data from csv files into your data warehouse."""
     task = SeedTask(
@@ -544,6 +561,7 @@ def seed(ctx, **kwargs):
 @requires.project
 @requires.runtime_config
 @requires.manifest
+@requires.postflight
 def snapshot(ctx, **kwargs):
     """Execute snapshots defined in your project"""
     task = SnapshotTask(
@@ -584,6 +602,7 @@ def source(ctx, **kwargs):
 @requires.project
 @requires.runtime_config
 @requires.manifest
+@requires.postflight
 def freshness(ctx, **kwargs):
     """check the current freshness of the project's sources"""
     task = FreshnessTask(
@@ -631,6 +650,7 @@ def freshness(ctx, **kwargs):
 @requires.project
 @requires.runtime_config
 @requires.manifest
+@requires.postflight
 def test(ctx, **kwargs):
     """Runs tests on data in deployed models. Run this after `dbt run`"""
     task = TestTask(
diff --git a/core/dbt/cli/requires.py b/core/dbt/cli/requires.py
--- a/core/dbt/cli/requires.py
+++ b/core/dbt/cli/requires.py
@@ -16,11 +16,34 @@
 from dbt.parser.manifest import ManifestLoader, write_manifest
 from dbt.profiler import profiler
 from dbt.tracking import active_user, initialize_from_flags, track_run
-from dbt.utils import cast_dict_to_dict_of_strings
+from dbt.utils import cast_dict_to_dict_of_strings, ExitCodes
 
 from click import Context
+from click.exceptions import ClickException
 from functools import update_wrapper
 import time
+import traceback
+
+
+class HandledExit(ClickException):
+    def __init__(self, result, success, exit_code: ExitCodes) -> None:
+        self.result = result
+        self.success = success
+        self.exit_code = exit_code
+
+    def show(self):
+        pass
+
+
+class UnhandledExit(ClickException):
+    exit_code = ExitCodes.UnhandledError.value
+
+    def __init__(self, exception: Exception, message: str) -> None:
+        self.exception = exception
+        self.message = message
+
+    def format_message(self) -> str:
+        return self.message
 
 
 def preflight(func):
@@ -61,11 +84,22 @@ def wrapper(*args, **kwargs):
         # Adapter management
         ctx.with_resource(adapter_management())
 
+        return func(*args, **kwargs)
+
+    return update_wrapper(wrapper, func)
+
+
+def postflight(func):
+    def wrapper(*args, **kwargs):
+        ctx = args[0]
         start_func = time.perf_counter()
+        success = False
 
         try:
-            (results, success) = func(*args, **kwargs)
-
+            result, success = func(*args, **kwargs)
+        except Exception as e:
+            raise UnhandledExit(e, message=traceback.format_exc())
+        finally:
             fire_event(
                 CommandCompleted(
                     command=ctx.command_path,
@@ -74,20 +108,15 @@ def wrapper(*args, **kwargs):
                     elapsed=time.perf_counter() - start_func,
                 )
             )
-        # Bare except because we really do want to catch ALL exceptions,
-        # i.e. we want to fire this event in ALL cases.
-        except:  # noqa
-            fire_event(
-                CommandCompleted(
-                    command=ctx.command_path,
-                    success=False,
-                    completed_at=get_json_string_utcnow(),
-                    elapsed=time.perf_counter() - start_func,
-                )
+
+        if not success:
+            raise HandledExit(
+                result=result,
+                success=success,
+                exit_code=ExitCodes.ModelError.value,
             )
-            raise
 
-        return (results, success)
+        return (result, success)
 
     return update_wrapper(wrapper, func)
 


Test patch:
diff --git a/tests/functional/cli/test_cli_exit_codes.py b/tests/functional/cli/test_cli_exit_codes.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/cli/test_cli_exit_codes.py
@@ -0,0 +1,38 @@
+import pytest
+
+from dbt.cli.main import cli
+from dbt.cli.requires import HandledExit
+
+
+good_sql = """
+select 1 as fun
+"""
+
+bad_sql = """
+someting bad
+"""
+
+
+class CliRunnerBase:
+    def run_cli(self):
+        ctx = cli.make_context(cli.name, ["run"])
+        return cli.invoke(ctx)
+
+
+class TestExitCodeZero(CliRunnerBase):
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {"model_one.sql": good_sql}
+
+    def test_no_exc_thrown(self, project):
+        self.run_cli()
+
+
+class TestExitCodeOne(CliRunnerBase):
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {"model_one.sql": bad_sql}
+
+    def test_exc_thrown(self, project):
+        with pytest.raises(HandledExit):
+            self.run_cli()
diff --git a/tests/functional/cli/test_error_handling.py b/tests/functional/cli/test_error_handling.py
new file mode 100644
--- /dev/null
+++ b/tests/functional/cli/test_error_handling.py
@@ -0,0 +1,17 @@
+import pytest
+
+from dbt.tests.util import run_dbt
+
+
+model_one_sql = """
+someting bad
+"""
+
+
+class TestHandledExit:
+    @pytest.fixture(scope="class")
+    def models(self):
+        return {"model_one.sql": model_one_sql}
+
+    def test_failed_run_does_not_throw(self, project):
+        run_dbt(["run"], expect_pass=False)