Repository: dbt-labs/dbt-core

Issue:
FreshnessStatus.Error causes exit status of 1 but no error level logging.
### Describe the bug
When I run `dbt --log-format=json source snapshot-freshness` and there is a `SourceFreshnessResult` with  `status = FreshnessStatus.Error: 'error'`, the log level of the output is `INFO` but message has a red `ERROR` and the cli returns with status code 1.

### Steps To Reproduce
As far as I can tell this will happen with any stale snapshot when you run `dbt --log-format=json source snapshot-freshness`.

### Expected behavior
I expect the log level of the output to be error.

### Screenshots and log output
```
{
  "timestamp": "2020-08-03T19:16:19.227088Z", 
  "message": "15:16:19 | 1 of 1 ERROR STALE freshness of foo.bar.............. [\u001b[31mERROR STALE\u001b[0m in 2.43s]", 
  "channel": "dbt", "level": 11,
  "levelname": "INFO",    <------------ ???
  "thread_name": "Thread-1", 
  "process": 65779, 
  "extra": {"unique_id":  "source.db.sch.tab", "run_state": "running"}
}
```

### System information
**Which database are you using dbt with?**
- [ ] postgres
- [x] redshift
- [ ] bigquery
- [ ] snowflake
- [ ] other (specify: ____________)


**The output of `dbt --version`:**
```
installed version: 0.16.1
   latest version: 0.17.2
```

**The operating system you're using:** MacOS Catalina

**The output of `python --version`:** 3.8.1

### Additional context
We parse the json output of a dbt run and report any ERROR level log statements to a monitoring tool, so this causes us to have a gap in our reporting.

Please let me know if I've mistaken a feature for a bug. Happy to contribute a fix if it is something that needs fixing.


Discussion:
@wrgoldstein Thanks for the detailed writeup. I think there's a reason for the behavior here, but I'm not convinced that it's how we want it to be in the long run.

In `dbt source snapshot-freshness`:
- dbt returns logs with `{"level": 14, "levelname": "ERROR"}` if it runs an invalid query as part of source freshness checks, e.g. if the `loaded_at_field` is misspelled
- dbt returns logs with `{"level": 11, "levelname": "INFO"}` and message `ERROR STALE`. In `sources.json`, it will report `"state": "error"`. (We want to rename this to `status` per the proposal in #2493).

Compare that to `dbt test`:
- dbt returns logs with `{"level": 14, "levelname": "ERROR"}` if a test errors when running (invalid SQL) OR if the test runs successfully and returns >0 results (FAIL)

@beckjake I'm inclined to say that `snapshot-freshness` should return level-13 (WARNING) and level-14 (ERROR) log output for sources that are beyond those respective thresholds. What do you think?
I don't have any feelings either way. Historically, dbt's stance has been that there are really only two levels: `debug`, and `info` - `debug` goes into the log file, and everything `info` and up also goes to stdout. I guess now that we show log levels we could also make changes here. It's buried behind a flag so I think it's hard to justify spending all that much energy on it.

I am in favor of accepting PRs that fix this, and this is probably a "good first issue".
Hello, I would like to give this a try as a first contribution. Unless you're already on it, @wrgoldstein ?

Just to be sure I'm going the right way, it seems to me that I should make `printer.print_fancy_output_line` able to call `logger.warning` and `logger.error`, depending on its `status` argument. Is that correct?
I'm not! Go for it :)

Resolving patch:
diff --git a/core/dbt/task/printer.py b/core/dbt/task/printer.py
--- a/core/dbt/task/printer.py
+++ b/core/dbt/task/printer.py
@@ -1,4 +1,4 @@
-from typing import Dict, Optional, Tuple
+from typing import Dict, Optional, Tuple, Callable
 from dbt.logger import (
     GLOBAL_LOGGER as logger,
     DbtStatusMessage,
@@ -13,8 +13,9 @@
 
 
 def print_fancy_output_line(
-        msg: str, status: str, index: Optional[int], total: Optional[int],
-        execution_time: Optional[float] = None, truncate: bool = False
+        msg: str, status: str, logger_fn: Callable, index: Optional[int],
+        total: Optional[int], execution_time: Optional[float] = None,
+        truncate: bool = False
 ) -> None:
     if index is None or total is None:
         progress = ''
@@ -39,7 +40,7 @@ def print_fancy_output_line(
     output = "{justified} [{status}{status_time}]".format(
         justified=justified, status=status, status_time=status_time)
 
-    logger.info(output)
+    logger_fn(output)
 
 
 def get_counts(flat_nodes) -> str:
@@ -63,12 +64,13 @@ def get_counts(flat_nodes) -> str:
 
 def print_start_line(description: str, index: int, total: int) -> None:
     msg = "START {}".format(description)
-    print_fancy_output_line(msg, 'RUN', index, total)
+    print_fancy_output_line(msg, 'RUN', logger.info, index, total)
 
 
 def print_hook_start_line(statement: str, index: int, total: int) -> None:
     msg = 'START hook: {}'.format(statement)
-    print_fancy_output_line(msg, 'RUN', index, total, truncate=True)
+    print_fancy_output_line(
+        msg, 'RUN', logger.info, index, total, truncate=True)
 
 
 def print_hook_end_line(
@@ -76,7 +78,7 @@ def print_hook_end_line(
 ) -> None:
     msg = 'OK hook: {}'.format(statement)
     # hooks don't fail into this path, so always green
-    print_fancy_output_line(msg, ui.green(status), index, total,
+    print_fancy_output_line(msg, ui.green(status), logger.info, index, total,
                             execution_time=execution_time, truncate=True)
 
 
@@ -84,23 +86,28 @@ def print_skip_line(
     model, schema: str, relation: str, index: int, num_models: int
 ) -> None:
     msg = 'SKIP relation {}.{}'.format(schema, relation)
-    print_fancy_output_line(msg, ui.yellow('SKIP'), index, num_models)
+    print_fancy_output_line(
+        msg, ui.yellow('SKIP'), logger.info, index, num_models)
 
 
 def print_cancel_line(model) -> None:
     msg = 'CANCEL query {}'.format(model)
-    print_fancy_output_line(msg, ui.red('CANCEL'), index=None, total=None)
+    print_fancy_output_line(
+        msg, ui.red('CANCEL'), logger.error, index=None, total=None)
 
 
-def get_printable_result(result, success: str, error: str) -> Tuple[str, str]:
+def get_printable_result(
+        result, success: str, error: str) -> Tuple[str, str, Callable]:
     if result.error is not None:
         info = 'ERROR {}'.format(error)
         status = ui.red(result.status)
+        logger_fn = logger.error
     else:
         info = 'OK {}'.format(success)
         status = ui.green(result.status)
+        logger_fn = logger.info
 
-    return info, status
+    return info, status, logger_fn
 
 
 def print_test_result_line(
@@ -111,24 +118,26 @@ def print_test_result_line(
     if result.error is not None:
         info = "ERROR"
         color = ui.red
+        logger_fn = logger.error
     elif result.status == 0:
         info = 'PASS'
         color = ui.green
-
+        logger_fn = logger.info
     elif result.warn:
         info = 'WARN {}'.format(result.status)
         color = ui.yellow
-
+        logger_fn = logger.warning
     elif result.fail:
         info = 'FAIL {}'.format(result.status)
         color = ui.red
-
+        logger_fn = logger.error
     else:
         raise RuntimeError("unexpected status: {}".format(result.status))
 
     print_fancy_output_line(
         "{info} {name}".format(info=info, name=model.name),
         color(info),
+        logger_fn,
         index,
         total,
         result.execution_time)
@@ -137,11 +146,13 @@ def print_test_result_line(
 def print_model_result_line(
     result, description: str, index: int, total: int
 ) -> None:
-    info, status = get_printable_result(result, 'created', 'creating')
+    info, status, logger_fn = get_printable_result(
+        result, 'created', 'creating')
 
     print_fancy_output_line(
         "{info} {description}".format(info=info, description=description),
         status,
+        logger_fn,
         index,
         total,
         result.execution_time)
@@ -152,7 +163,8 @@ def print_snapshot_result_line(
 ) -> None:
     model = result.node
 
-    info, status = get_printable_result(result, 'snapshotted', 'snapshotting')
+    info, status, logger_fn = get_printable_result(
+        result, 'snapshotted', 'snapshotting')
     cfg = model.config.to_dict()
 
     msg = "{info} {description}".format(
@@ -160,6 +172,7 @@ def print_snapshot_result_line(
     print_fancy_output_line(
         msg,
         status,
+        logger_fn,
         index,
         total,
         result.execution_time)
@@ -168,7 +181,7 @@ def print_snapshot_result_line(
 def print_seed_result_line(result, schema_name: str, index: int, total: int):
     model = result.node
 
-    info, status = get_printable_result(result, 'loaded', 'loading')
+    info, status, logger_fn = get_printable_result(result, 'loaded', 'loading')
 
     print_fancy_output_line(
         "{info} seed file {schema}.{relation}".format(
@@ -176,6 +189,7 @@ def print_seed_result_line(result, schema_name: str, index: int, total: int):
             schema=schema_name,
             relation=model.alias),
         status,
+        logger_fn,
         index,
         total,
         result.execution_time)
@@ -185,15 +199,19 @@ def print_freshness_result_line(result, index: int, total: int) -> None:
     if result.error:
         info = 'ERROR'
         color = ui.red
+        logger_fn = logger.error
     elif result.status == 'error':
         info = 'ERROR STALE'
         color = ui.red
+        logger_fn = logger.error
     elif result.status == 'warn':
         info = 'WARN'
         color = ui.yellow
+        logger_fn = logger.warning
     else:
         info = 'PASS'
         color = ui.green
+        logger_fn = logger.info
 
     if hasattr(result, 'node'):
         source_name = result.node.source_name
@@ -211,6 +229,7 @@ def print_freshness_result_line(result, index: int, total: int) -> None:
     print_fancy_output_line(
         msg,
         color(info),
+        logger_fn,
         index,
         total,
         execution_time=result.execution_time
@@ -297,7 +316,8 @@ def print_skip_caused_by_error(
 ) -> None:
     msg = ('SKIP relation {}.{} due to ephemeral model error'
            .format(schema, relation))
-    print_fancy_output_line(msg, ui.red('ERROR SKIP'), index, num_models)
+    print_fancy_output_line(
+        msg, ui.red('ERROR SKIP'), logger.error, index, num_models)
     print_run_result_error(result, newline=False)
 
 


Test patch: