Repository: dbt-labs/dbt-core

Issue:
[CT-2063] [Regression] Console Logging in dbt 1.4.1 is not interactive
### Is this a regression in a recent version of dbt-core?

- [X] I believe this is a regression in dbt-core functionality
- [X] I have searched the existing issues, and I could not find an existing issue for this regression

### Current Behavior

We recently upgraded to dbt 1.4.1 and noticed the console logging (both locally and when we run commands via Airflow) is not interactive/real-time anymore.

Notice the dbt command started running at 8:01 but didn't log until 9:27 when it was fully complete.

```
2023-02-08, 08:01:12 UTC] {dbt_hook.py:137} INFO - dbt run --project-dir /tmp/dbt --profiles-dir /tmp/dbt/profile --target prod --models analytics_tables
[2023-02-08, 08:01:12 UTC] {dbt_hook.py:150} INFO - Output:
[2023-02-08, 09:27:22 UTC] {dbt_hook.py:154} INFO - [0m08:01:15  Running with dbt=1.4.1
[2023-02-08, 09:27:22 UTC] {dbt_hook.py:154} INFO - [0m08:03:56  Found 100 models, 100 tests, 26 snapshots, 0 analyses, 100 macros, 4 operations, 90 seed files, 100 sources, 0 exposures, 0 metrics
[2023-02-08, 09:27:22 UTC] {dbt_hook.py:154} INFO - [0m08:07:14  Running 1 on-run-start hook
[2023-02-08, 09:27:22 UTC] {dbt_hook.py:154} INFO - [0m08:07:14  1 of 1 START hook: elementary.on-run-start.0 ................................... [RUN]
[2023-02-08, 09:27:22 UTC] {dbt_hook.py:154} INFO - [0m08:07:14  1 of 1 OK hook: elementary.on-run-start.0 ...................................... [[32mOK[0m in 0.00s]
[2023-02-08, 09:27:22 UTC] {dbt_hook.py:154} INFO - [0m08:07:14  Concurrency: 16 threads (target='prod')
[2023-02-08, 09:27:22 UTC] {dbt_hook.py:154} INFO - [0m08:07:14  1 of 27 START sql table model transform.table [RUN]
```

### Expected/Previous Behavior

This is how it logged previously in real-time

```
[2023-02-07, 08:02:32 UTC] {dbt_hook.py:137} INFO - dbt run --project-dir /tmp/dbt --profiles-dir /tmp/dbt/profile --target prod --models analytics_tables
[2023-02-07, 08:02:32 UTC] {dbt_hook.py:150} INFO - Output:
[2023-02-07, 08:02:35 UTC] {dbt_hook.py:154} INFO - [0m08:02:35  Running with dbt=1.3.1
[2023-02-07, 08:05:27 UTC] {dbt_hook.py:154} INFO - [0m08:05:27  Found 100 models, 100 tests, 26 snapshots, 0 analyses, 100 macros, 4 operations, 90 seed files, 100 sources, 0 exposures, 0 metrics
[2023-02-07, 08:09:23 UTC] {dbt_hook.py:154} INFO - [0m08:09:23  Running 1 on-run-start hook
[2023-02-07, 08:09:23 UTC] {dbt_hook.py:154} INFO - [0m08:09:23  1 of 1 START hook: elementary.on-run-start.0 ................................... [RUN]
[2023-02-07, 08:09:23 UTC] {dbt_hook.py:154} INFO - [0m08:09:23  1 of 1 OK hook: elementary.on-run-start.0 ...................................... [[32mOK[0m in 0.00s]
[2023-02-07, 08:09:23 UTC] {dbt_hook.py:154} INFO - [0m08:09:23  Concurrency: 16 threads (target='prod')
[2023-02-07, 08:09:23 UTC] {dbt_hook.py:154} INFO - [0m08:09:23  1 of 27 START sql table model transform.table .... [RUN]

### Steps To Reproduce

All we did was upgrade `dbt-core` and `dbt-snowflake` to the latest versions

### Relevant log output

_No response_

### Environment

```markdown
- OS:
- Python:
- dbt (working version): 1.3.1
- dbt (regression version): 1.4.1
```


### Which database adapter are you using with dbt?

snowflake

### Additional Context

_No response_


Discussion:
There were significant revisions to dbt's logging infrastructure in 1.4, including to the mechanism for writing to stdout. We previously used Python's logging.StreamHandler class, but now we're writing to stdout directly. Unlike the StreamHandler implementation, we do not call flush() after every log entry is written. 

So we have a strong lead on he root cause, however I have not been able to reproduce the behavior.

@ajbosco Could you let us know what OS you are on, and the nature of the dbt_hook.py wrapper it looks like you are using?

In the meantime, I will prepare a changeset which adds the flush() after every write. Not flushing between writes was definitely an oversight.


@jtcohen6 Does that sound like the right approach to you?
@peterallenwebb That sounds like a good approach to me! Thanks for digging into this, and coming up with a probable lead.

Resolving patch:
diff --git a/core/dbt/events/eventmgr.py b/core/dbt/events/eventmgr.py
--- a/core/dbt/events/eventmgr.py
+++ b/core/dbt/events/eventmgr.py
@@ -88,41 +88,40 @@ def __init__(self, event_manager: "EventManager", config: LoggerConfig) -> None:
         self.level: EventLevel = config.level
         self.event_manager: EventManager = event_manager
         self._python_logger: Optional[logging.Logger] = config.logger
-        self._stream: Optional[TextIO] = config.output_stream
+
+        if config.output_stream is not None:
+            stream_handler = logging.StreamHandler(config.output_stream)
+            self._python_logger = self._get_python_log_for_handler(stream_handler)
 
         if config.output_file_name:
-            log = logging.getLogger(config.name)
-            log.setLevel(_log_level_map[config.level])
-            handler = RotatingFileHandler(
+            file_handler = RotatingFileHandler(
                 filename=str(config.output_file_name),
                 encoding="utf8",
                 maxBytes=10 * 1024 * 1024,  # 10 mb
                 backupCount=5,
             )
+            self._python_logger = self._get_python_log_for_handler(file_handler)
 
-            handler.setFormatter(logging.Formatter(fmt="%(message)s"))
-            log.handlers.clear()
-            log.addHandler(handler)
-
-            self._python_logger = log
+    def _get_python_log_for_handler(self, handler: logging.Handler):
+        log = logging.getLogger(self.name)
+        log.setLevel(_log_level_map[self.level])
+        handler.setFormatter(logging.Formatter(fmt="%(message)s"))
+        log.handlers.clear()
+        log.addHandler(handler)
+        return log
 
     def create_line(self, msg: EventMsg) -> str:
         raise NotImplementedError()
 
     def write_line(self, msg: EventMsg):
         line = self.create_line(msg)
-        python_level = _log_level_map[EventLevel(msg.info.level)]
         if self._python_logger is not None:
             send_to_logger(self._python_logger, msg.info.level, line)
-        elif self._stream is not None and _log_level_map[self.level] <= python_level:
-            self._stream.write(line + "\n")
 
     def flush(self):
         if self._python_logger is not None:
             for handler in self._python_logger.handlers:
                 handler.flush()
-        elif self._stream is not None:
-            self._stream.flush()
 
 
 class _TextLogger(_Logger):


Test patch: