Repository: PrefectHQ/prefect

Issue:
Add Flow and Task identifying information to LogRecords
## Current behavior
- All log entries have the default [LogRecord attributes](https://docs.python.org/3/library/logging.html#logrecord-objects)... 
- There is no universally identifiable flow slug

## Proposed behavior
...It would be useful to attach Core-specific information to log entries:
- `task_name` & `task_slug`: available only on entries emitted from Tasks and TaskRunners
- `flow_name` & `flow_slug`: available only on entries emitted from Flows and FlowRunners

The addition of the `flow_slug` makes is possible to uniquely identify flows other than by name.

## Example
One can now take a single stream of log entries during a local flow run and separate it by entity without parsing the `LogRecord.msg` field (which is unreliable). This makes is possible to build richer feedback interfaces for filtering, organizing, and otherwise selectively manipulating log entries.

Here is an example CLI interface that would be made possible by theses features:

![local-progress](https://user-images.githubusercontent.com/590471/72913913-dbd00880-3d0b-11ea-8c85-a0e74e1c9551.gif)

(Given the following flow):
```python
from utils import seed, random_data
import prefect

@prefect.task
def extract():
    return seed.get()

@prefect.task
def transform(seed):
    logger = prefect.context.get("logger")
    data = []
    for x in range(10):
        c_data = random_data(seed)
        logger.info("debug info: {}".format(c_data))
        data.append(c_data)
    return data

@prefect.task
def load(data):
    # do something with the data....
    pass

with prefect.Flow("ETL") as flow:
    e = extract()
    t = transform(e)
    load(t)
```



Discussion:
Here is a concept implementation: https://github.com/PrefectHQ/prefect/commit/b075f14151ec37352cc5bcc7ad104c56af473bdd

... though it has a few drawbacks:
- mypy doesn't like having either a `LoggerAdapter` or a `Logger` as class members
- pulling the logger from within a `Task`s run body leads in a different `Task.slug` bring attached to the resulting LogRecord (if automatically generated) than what is observed by records from a TaskRunner. I have yet to explain this behavior.
I think you might be able to do this much simpler through the use of a Logger Record Factory + extracting this information from context; we used to do something like this, I'll try to dig up the old code.
Simpler would be preferable --very interested in any old code dug up. 

I'm using the LogRecordFactory now, however, when I tried to pull this from the global `prefect.context` I wasn't successful. I also worried that if I were to add these values dynamically to context during the run while tasks are running concurrently, then overlapping context keys would be in contention, resulting in inconsistent values (that is, a task may log with another task slug). (I do realize that tasks don't run in parallel by default unless submitted to an executor that would leverage threads or processes to do so)

The approach attempted here is to use an adapter as a sort of context: anything logging though these adapters will always append extra information to the proper logs with the supported `extra` dictionary.
@wagoodman here is the record factory we used to have: https://github.com/PrefectHQ/prefect/blob/99f5437ed21425461eb9b1dc58e27f7ac2afc028/src/prefect/utilities/logging.py#L44-L51

> I also worried that if I were to add these values dynamically to context during the run while tasks are running concurrently...

We actually modify context for all runs right now.  Because context is `thread.local` and because of how all our executors work, as long as you modify context within the `initialize_run` method of the relevant runner objects you should be good to go!

Resolving patch:
diff --git a/src/prefect/utilities/logging.py b/src/prefect/utilities/logging.py
--- a/src/prefect/utilities/logging.py
+++ b/src/prefect/utilities/logging.py
@@ -23,6 +23,17 @@
 from prefect.utilities.context import context
 
 
+_original_log_record_factory = logging.getLogRecordFactory()
+
+PREFECT_LOG_RECORD_ATTRIBUTES = (
+    "flow_name",
+    "flow_run_id",
+    "task_name",
+    "task_slug",
+    "task_run_id",
+)
+
+
 class CloudHandler(logging.StreamHandler):
     def __init__(self) -> None:
         super().__init__(sys.stdout)
@@ -151,6 +162,27 @@ def _make_error_log(self, message: str) -> dict:
         return log
 
 
+def _log_record_context_injector(*args: Any, **kwargs: Any) -> logging.LogRecord:
+    """
+    A custom logger LogRecord Factory that injects selected context parameters into newly created logs.
+
+    Args:
+        - *args: arguments to pass to the original LogRecord Factory
+        - **kwargs: keyword arguments to pass to the original LogRecord Factory
+
+    Returns:
+        - logging.LogRecord: the newly created LogRecord
+    """
+    record = _original_log_record_factory(*args, **kwargs)
+
+    for attr in PREFECT_LOG_RECORD_ATTRIBUTES:
+        value = prefect.context.get(attr, None)
+        if value:
+            setattr(record, attr, value)
+
+    return record
+
+
 def configure_logging(testing: bool = False) -> logging.Logger:
     """
     Creates a "prefect" root logger with a `StreamHandler` that has level and formatting
@@ -164,6 +196,8 @@ def configure_logging(testing: bool = False) -> logging.Logger:
     Returns:
         - logging.Logger: a configured logging object
     """
+    logging.setLogRecordFactory(_log_record_context_injector)
+
     name = "prefect-test-logger" if testing else "prefect"
     logger = logging.getLogger(name)
     handler = logging.StreamHandler(sys.stdout)
@@ -194,6 +228,7 @@ def get_logger(name: str = None) -> logging.Logger:
     Returns:
         - logging.Logger: a configured logging object with the appropriate name
     """
+
     if name is None:
         return prefect_logger
     else:


Test patch:
diff --git a/tests/utilities/test_logging.py b/tests/utilities/test_logging.py
--- a/tests/utilities/test_logging.py
+++ b/tests/utilities/test_logging.py
@@ -247,3 +247,65 @@ def test_get_logger_with_name_returns_child_logger():
 
     assert prefect_logger is child_logger
     assert prefect_logger is logging.getLogger("prefect").getChild("test")
+
+
+def test_context_attributes():
+    items = {
+        "flow_run_id": "fri",
+        "flow_name": "fn",
+        "task_run_id": "tri",
+        "task_name": "tn",
+        "task_slug": "ts",
+    }
+
+    class DummyFilter(logging.Filter):
+        called = False
+
+        def filter(self, record):
+            self.called = True
+            for k, v in items.items():
+                assert getattr(record, k, None) == v
+
+    test_filter = DummyFilter()
+    logger = logging.getLogger("prefect")
+    logger.addFilter(test_filter)
+
+    with context(items):
+        logger.info("log entry!")
+
+    logger.filters.pop()
+
+    assert test_filter.called
+
+
+def test_context_only_specified_attributes():
+    items = {
+        "flow_run_id": "fri",
+        "flow_name": "fn",
+        "task_run_id": "tri",
+    }
+
+    class DummyFilter(logging.Filter):
+        called = False
+
+        def filter(self, record):
+            self.called = True
+            for k, v in items.items():
+                assert getattr(record, k, None) == v
+
+            not_expected = set(utilities.logging.PREFECT_LOG_RECORD_ATTRIBUTES) - set(
+                items.keys()
+            )
+            for key in not_expected:
+                assert key not in record.__dict__.keys()
+
+    test_filter = DummyFilter()
+    logger = logging.getLogger("prefect")
+    logger.addFilter(test_filter)
+
+    with context(items):
+        logger.info("log entry!")
+
+    logger.filters.pop()
+
+    assert test_filter.called