Repository: PrefectHQ/prefect

Issue:
Add optional stdout logging for tasks
## Current behavior
Currently, `print` statements or any other stdout is not treated in any special way.  This has two consequences:

- users who use `print` statements locally and can see the output are surprised when the output doesn't appear in the Cloud logs
- print statements from third-party libraries that might be useful are not surfaced in cloud execution

## Proposed behavior
We should introduce a new keyword argument on all Tasks `log_stdout` which defaults to `True`.  When `True`, all standard out is converted to a DEBUG level log.  When toggled to `False`, we will not redirect.

This will capture richer information from task runs, but also allow for incredibly simple logging for our less advanced users:

```python
@task
def example():
    print('hi')
    print('this is a test')
    print('of our emergency broadcast system')
```

will result in 3 debug logs.

## Example
This is actually very simple with Python builtins:

```python
import prefect
from contextlib import redirect_stdout

class RedirectToLog:
    def write(self, s):
        # avoids blank newlines
        if s.strip():
            prefect.utilities.logging.prefect_logger.debug(s)

to_debug_log = RedirectToLog()

with redirect_stdout(to_debug_log):
    print('hi')
    print('this is a test')
    print('of our emergency broadcast system')
```

results in :

```
[2020-02-17 00:02:13,642] 4-- DEBUG - prefect | hi
[2020-02-17 00:02:13,642] 4-- DEBUG - prefect | this is a test
[2020-02-17 00:02:13,643] 4-- DEBUG - prefect | of our emergency broadcast system
```

**NOTE:** In our case, we want want to generate the redirect class on the fly so that it uses the task run specific logger in each case.


Discussion:
I like where this is going, I think there should be discussion on a few points:
- Would we want this to be opt-in or opt-out behavior?
- Given that prefect is a lib, having global side-effects on stdout may be undesirable/unexpected. The docs also [call this out explicitly](https://docs.python.org/3/library/contextlib.html#contextlib.redirect_stdout). (Notable enough to discuss and actively acknowledge in advance.)
- Are we interested in also capturing stderr (perhaps under the ERROR logging level)?
- This [will not capture stdout from lower-level calls](https://bugs.python.org/issue24500) (e.g. prints from C extensions), is this something that we're interested in also capturing?
- as mentioned in the description, I vote opt-in because one of the main purposes would be ease of use and so far I've found that users have expected this already
- Hm I don't think our usage would fully classify as a global side effect, especially since we are acting as a manager of the task run; however, the callout about use in threaded applications is a little iffy - I agree this warrants further investigation / discussion
- this would cause weird recursive issues with logging that I'd rather avoid (since most logs go to stderr)
- that's fine, most of the output we want to capture is basic print statements 

Resolving patch:
diff --git a/src/prefect/core/task.py b/src/prefect/core/task.py
--- a/src/prefect/core/task.py
+++ b/src/prefect/core/task.py
@@ -154,6 +154,8 @@ def run(self, x, y):
             result of the previous handler.
         - on_failure (Callable, optional): A function with signature `fn(task: Task, state: State) -> None`
             with will be called anytime this Task enters a failure state
+        - log_stdout (bool, optional): Toggle whether or not to send stdout messages to
+            the Prefect logger. Defaults to `False`.
 
     Raises:
         - TypeError: if `tags` is of type `str`
@@ -180,8 +182,8 @@ def __init__(
         result_handler: "ResultHandler" = None,
         state_handlers: List[Callable] = None,
         on_failure: Callable = None,
+        log_stdout: bool = False,
     ):
-
         self.name = name or type(self).__name__
         self.slug = slug or str(uuid.uuid4())
 
@@ -255,6 +257,8 @@ def __init__(
             )
         self.auto_generated = False
 
+        self.log_stdout = log_stdout
+
     def __repr__(self) -> str:
         return "<Task: {self.name}>".format(self=self)
 
diff --git a/src/prefect/engine/task_runner.py b/src/prefect/engine/task_runner.py
--- a/src/prefect/engine/task_runner.py
+++ b/src/prefect/engine/task_runner.py
@@ -1,4 +1,5 @@
 import copy
+from contextlib import redirect_stdout
 import itertools
 from typing import (
     TYPE_CHECKING,
@@ -873,9 +874,16 @@ def get_task_run_state(
                 timeout_handler or prefect.utilities.executors.timeout_handler
             )
             raw_inputs = {k: r.value for k, r in inputs.items()}
-            result = timeout_handler(
-                self.task.run, timeout=self.task.timeout, **raw_inputs
-            )
+
+            if self.task.log_stdout:
+                with redirect_stdout(prefect.utilities.logging.RedirectToLog(self.logger)):  # type: ignore
+                    result = timeout_handler(
+                        self.task.run, timeout=self.task.timeout, **raw_inputs
+                    )
+            else:
+                result = timeout_handler(
+                    self.task.run, timeout=self.task.timeout, **raw_inputs
+                )
 
         except KeyboardInterrupt:
             self.logger.debug("Interrupt signal raised, cancelling task run.")
diff --git a/src/prefect/utilities/logging.py b/src/prefect/utilities/logging.py
--- a/src/prefect/utilities/logging.py
+++ b/src/prefect/utilities/logging.py
@@ -258,3 +258,34 @@ def get_logger(name: str = None) -> logging.Logger:
         return prefect_logger
     else:
         return prefect_logger.getChild(name)
+
+
+class RedirectToLog:
+    """
+    Custom redirect of stdout messages to logs
+
+    Args:
+        - logger (logging.Logger, optional): an optional logger to redirect stdout. If
+            not provided a logger names `stdout` will be created.
+    """
+
+    def __init__(self, logger: logging.Logger = None) -> None:
+        self.stdout_logger = logger or get_logger("stdout")
+
+    def write(self, s: str) -> None:
+        """
+        Write message from stdout to a prefect logger.
+        Note: blank newlines will not be logged.
+
+        Args:
+            s (str): the message from stdout to be logged
+        """
+        if s.strip():
+            self.stdout_logger.info(s)
+
+    def flush(self) -> None:
+        """
+        Implemented flush operation for logger handler
+        """
+        for handler in self.stdout_logger.handlers:
+            handler.flush()


Test patch:
diff --git a/tests/core/test_task.py b/tests/core/test_task.py
--- a/tests/core/test_task.py
+++ b/tests/core/test_task.py
@@ -262,6 +262,13 @@ def test_create_task_with_and_without_checkpoint(self):
         s = Task(checkpoint=True)
         assert s.checkpoint is True
 
+    def test_create_task_with_and_without_log_stdout(self):
+        t = Task()
+        assert t.log_stdout is False
+
+        s = Task(log_stdout=True)
+        assert s.log_stdout is True
+
 
 def test_task_has_logger():
     t = Task()
diff --git a/tests/engine/test_task_runner.py b/tests/engine/test_task_runner.py
--- a/tests/engine/test_task_runner.py
+++ b/tests/engine/test_task_runner.py
@@ -2242,3 +2242,27 @@ def t(x):
     state = TaskRunner(task=t).run(upstream_states=upstream_states)
     assert state.is_successful()
     assert state.result == "cool"
+
+
+def test_task_runner_logs_stdout(caplog):
+    class MyTask(Task):
+        def run(self):
+            print("TEST_HERE")
+
+    task = MyTask(log_stdout=True)
+    TaskRunner(task=task).run()
+
+    logs = [r.message for r in caplog.records]
+    assert logs[1] == "TEST_HERE"
+
+
+def test_task_runner_logs_stdout_disabled(caplog):
+    class MyTask(Task):
+        def run(self):
+            print("TEST_HERE")
+
+    task = MyTask()
+    TaskRunner(task=task).run()
+
+    logs = [r.message for r in caplog.records]
+    assert "TEST_HERE" not in logs
diff --git a/tests/utilities/test_logging.py b/tests/utilities/test_logging.py
--- a/tests/utilities/test_logging.py
+++ b/tests/utilities/test_logging.py
@@ -310,8 +310,6 @@ def filter(self, record):
 
     assert test_filter.called
 
-
-def test_extra_loggers():
     with utilities.configuration.set_temporary_config(
         {"logging.extra_loggers": ["extra_logger"]}
     ):
@@ -319,3 +317,16 @@ def test_extra_loggers():
         assert (
             type(logging.root.manager.loggerDict.get("extra_logger")) == logging.Logger
         )
+
+
+def test_redirect_to_log(caplog):
+    log_stdout = utilities.logging.RedirectToLog()
+
+    log_stdout.write("TEST1")
+    log_stdout.write("")
+    log_stdout.write("TEST2")
+
+    logs = [r.message for r in caplog.records if r.levelname == "INFO"]
+    assert logs == ["TEST1", "TEST2"]
+
+    log_stdout.flush()