Repository: PrefectHQ/prefect

Issue:
Custom logger raises exception outside a flow and shows duplicate logs inside a flow
### First check

- [X] I added a descriptive title to this issue.
- [X] I used the GitHub search to find a similar issue and didn't find it.
- [X] I searched the Prefect documentation for this issue.
- [X] I checked that this issue is related to Prefect and not one of its dependencies.

### Bug summary

 I created an own logger but have some problems with prefect to get it running. I only found documentation on how to set up a custom logger for v1: https://docs-v1.prefect.io/core/concepts/logging.html#extra-loggers
But I think it should work the same for v2. I also had a look here: https://discourse.prefect.io/t/prefect-logging-faq/1476

I also posted to the Slack channel first to make sure I don't miss to set something in the configs: https://prefect-community.slack.com/archives/CL09KU1K7/p1670795120210289

Using prefect >= 2.6.9 and calling `regular_log()` in the main function, I get an exception when trying to use my own logger. This also happens when I just define `import from prefect import flow` and have no flow defined.

Using prefect 2.6.8 and calling `regular_log()` it does not crash.

But in any cases I see duplicate logs calling `flow_log()`. Next to my own logger also prefect prints to logging.

### Reproduction


my_logger.py
```python3
import logging
import sys

logger = logging.getLogger("my-logger")
logger.setLevel("DEBUG")
log_stream = logging.StreamHandler(sys.stdout)
formatter = logging.Formatter("[%(levelname)s %(asctime)s %(filename)s:%(lineno)d] %(message)s")
log_stream.setFormatter(formatter)
logger.addHandler(log_stream)
```

my_flow.py
```python3
from prefect import flow
from my_logger import logger

def regular_log():
    print("Log inside a regular function")
    logger.debug("debug")
    logger.info("info")
    logger.error("error")
    logger.warning("warning")
    logger.critical("critical")

@flow(name="my-flow")
def flow_log():
    print("Log inside a prefect flow")
    logger.debug("debug")
    logger.info("info")
    logger.error("error")
    logger.warning("warning")
    logger.critical("critical")

if __name__ == "__main__":
    regular_log()
    print("-" * 30)
    flow_log()
```

I tried both:

`PREFECT_LOGGING_EXTRA_LOGGERS=my-logger python my_flow.py`
and
`prefect config set PREFECT_LOGGING_EXTRA_LOGGERS="my-logger"`
`python my_flow.py`


### Error

Using prefect >= 2.6.9
```python3
Log inside a regular function
Traceback (most recent call last):
  File "/Users/niconeumann/.pyenv/versions/3.9.13/lib/python3.9/site-packages/prefect/logging/handlers.py", line 278, in emit
    self.get_worker(profile).enqueue(self.prepare(record, profile.settings))
  File "/Users/niconeumann/.pyenv/versions/3.9.13/lib/python3.9/site-packages/prefect/logging/handlers.py", line 314, in prepare
    raise MissingContextError(
prefect.exceptions.MissingContextError: Logger 'my-logger' attempted to send logs to Orion without a flow run id. The Orion log handler can only send logs within flow run contexts unless the flow run id is manually provided.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/niconeumann/test/flows/my_flow.py", line 26, in <module>
    regular_log()
  File "/Users/niconeumann/test/flows/my_flow.py", line 8, in regular_log
    logger.debug("debug")
  File "/Users/niconeumann/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1434, in debug
    self._log(DEBUG, msg, args, **kwargs)
  File "/Users/niconeumann/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1589, in _log
    self.handle(record)
  File "/Users/niconeumann/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1599, in handle
    self.callHandlers(record)
  File "/Users/niconeumann/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 1661, in callHandlers
    hdlr.handle(record)
  File "/Users/niconeumann/.pyenv/versions/3.9.13/lib/python3.9/logging/__init__.py", line 952, in handle
    self.emit(record)
  File "/Users/niconeumann/.pyenv/versions/3.9.13/lib/python3.9/site-packages/prefect/logging/handlers.py", line 280, in emit
    self.handleError(record)
  File "/Users/niconeumann/.pyenv/versions/3.9.13/lib/python3.9/site-packages/prefect/logging/handlers.py", line 288, in handleError
    warnings.warn(exc, stacklevel=8)
TypeError: expected string or bytes-like object
```

Using prefect 2.6.8 or >= 2.6.9 and only calling `flow_log()`:
```python3
23:11:43.275 | INFO    | prefect.engine - Created flow run 'brown-anteater' for flow 'my-flow'
Log inside a prefect flow
[DEBUG 2022-12-11 23:11:43,345 my_flow.py:18] debug
23:11:43.345 | DEBUG   | my-logger - debug
[INFO 2022-12-11 23:11:43,346 my_flow.py:19] info
23:11:43.346 | INFO    | my-logger - info
[ERROR 2022-12-11 23:11:43,346 my_flow.py:20] error
23:11:43.346 | ERROR   | my-logger - error
[WARNING 2022-12-11 23:11:43,346 my_flow.py:21] warning
23:11:43.346 | WARNING | my-logger - warning
[CRITICAL 2022-12-11 23:11:43,346 my_flow.py:22] critical
23:11:43.346 | CRITICAL | my-logger - critical
23:11:43.356 | INFO    | Flow run 'brown-anteater' - Finished in state Completed()
```


### Versions


I tried 2.6.8, 2.6.9 and 2.7.1

```Text
Version:             2.7.1
API version:         0.8.3
Python version:      3.9.13
Git commit:          33f7de50
Built:               Thu, Dec 8, 2022 4:09 PM
OS/Arch:             darwin/arm64
Profile:             local
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.39.4
```

```Text
Version:             2.6.9
API version:         0.8.3
Python version:      3.9.13
Git commit:          014d093e
Built:               Tue, Nov 22, 2022 2:14 PM
OS/Arch:             darwin/arm64
Profile:             local
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.39.4
```

```Text
Version:             2.6.8
API version:         0.8.3
Python version:      3.9.13
Git commit:          68044e28
Built:               Thu, Nov 17, 2022 3:19 PM
OS/Arch:             darwin/arm64
Profile:             local
Server type:         ephemeral
Server:
  Database:          sqlite
  SQLite version:    3.39.4
```




Discussion:
Thanks for the report! 

If you do `log_stream = logging.StreamHandler(sys.stdout)` you are adding a handler in _addition_ to the Prefect console handler. That's the reason you have duplicate logs. You could probably disable propagation to avoid it being passed to the Prefect console handler as well.

The following error is expected, we want to avoid confusion of "why aren't my logs in Prefect Cloud" when called from outside of a flow / task. We may want to add a way to disable it, but I'm not sure what the best behavior is here.
```
prefect.exceptions.MissingContextError: Logger 'my-logger' attempted to send logs to Orion without a flow run id. The Orion log handler can only send logs within flow run contexts unless the flow run id is manually provided.
```

We should investigate this:
```
  File "/Users/niconeumann/.pyenv/versions/3.9.13/lib/python3.9/site-packages/prefect/logging/handlers.py", line 288, in handleError
    warnings.warn(exc, stacklevel=8)
TypeError: expected string or bytes-like object
```
I'm running into this issue with custom loggers and the `prefect.exceptions.MissingContextError` when logging outside of the flow run context.

I have a custom package which uses a package logger that logs some things at package import time, which occurs outside of the flow run context. With the current prefect logging implementation, this means I cannot include this logger in `PREFECT_LOGGING_EXTRA_LOGGERS`.  I'm on board with removing (or having a way to disable) this error because it seems to be blocking some common use cases.

Here's a reproducible example

With `test_flow.py` as:

```python
from prefect import flow
import logging

@flow
def hello_world():
    print('hello world!')

if __name__ == "__main__":

    import logging
    logger = logging.getLogger("custom")
    logger.setLevel("INFO")

    # This log statement will fail if PREFECT_LOGGING_EXTRA_LOGGERS="custom"
    logger.info("Initializing...")

    hello_world()
```

`python test.py` works, but `PREFECT_LOGGING_EXTRA_LOGGERS="custom" python flows/test.py` gives:

```
❯ PREFECT_LOGGING_EXTRA_LOGGERS="custom" python flows/test.py
Traceback (most recent call last):
  File "/Users/lmendelowitz/.local/share/virtualenvs/tempt-p2XF0II4/lib/python3.10/site-packages/prefect/logging/handlers.py", line 278, in emit
    self.get_worker(profile).enqueue(self.prepare(record, profile.settings))
  File "/Users/lmendelowitz/.local/share/virtualenvs/tempt-p2XF0II4/lib/python3.10/site-packages/prefect/logging/handlers.py", line 314, in prepare
    raise MissingContextError(
prefect.exceptions.MissingContextError: Logger 'custom' attempted to send logs to Orion without a flow run id. The Orion log handler can only send logs within flow run contexts unless the flow run id is manually provided.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/lmendelowitz/playground/temp/tempt/flows/test.py", line 17, in <module>
    logger.info("Initializing...")
  File "/Users/lmendelowitz/.pyenv/versions/3.10.8/lib/python3.10/logging/__init__.py", line 1477, in info
    self._log(INFO, msg, args, **kwargs)
  File "/Users/lmendelowitz/.pyenv/versions/3.10.8/lib/python3.10/logging/__init__.py", line 1624, in _log
    self.handle(record)
  File "/Users/lmendelowitz/.pyenv/versions/3.10.8/lib/python3.10/logging/__init__.py", line 1634, in handle
    self.callHandlers(record)
  File "/Users/lmendelowitz/.pyenv/versions/3.10.8/lib/python3.10/logging/__init__.py", line 1696, in callHandlers
    hdlr.handle(record)
  File "/Users/lmendelowitz/.pyenv/versions/3.10.8/lib/python3.10/logging/__init__.py", line 968, in handle
    self.emit(record)
  File "/Users/lmendelowitz/.local/share/virtualenvs/tempt-p2XF0II4/lib/python3.10/site-packages/prefect/logging/handlers.py", line 280, in emit
    self.handleError(record)
  File "/Users/lmendelowitz/.local/share/virtualenvs/tempt-p2XF0II4/lib/python3.10/site-packages/prefect/logging/handlers.py", line 288, in handleError
    warnings.warn(exc, stacklevel=8)
TypeError: expected string or bytes-like object
```

```
❯ prefect version
Version:             2.7.5
API version:         0.8.4
Python version:      3.10.8
Git commit:          5e0fa585
Built:               Thu, Dec 29, 2022 5:31 PM
OS/Arch:             darwin/arm64
Profile:             test
Server type:         cloud
```


Thanks for the additional details!

I'd accept a PR adding a setting for this `PREFECT_LOGGING_ORION_WHEN_MISSING_FLOW: Literal["warn", "error", "ignore"] = "warn"`

I'd also review a fix for the warning error.

Resolving patch:
diff --git a/src/prefect/logging/handlers.py b/src/prefect/logging/handlers.py
--- a/src/prefect/logging/handlers.py
+++ b/src/prefect/logging/handlers.py
@@ -26,6 +26,7 @@
     PREFECT_LOGGING_ORION_BATCH_SIZE,
     PREFECT_LOGGING_ORION_ENABLED,
     PREFECT_LOGGING_ORION_MAX_LOG_SIZE,
+    PREFECT_LOGGING_ORION_WHEN_MISSING_FLOW,
 )
 
 
@@ -282,11 +283,19 @@ def emit(self, record: logging.LogRecord):
     def handleError(self, record: logging.LogRecord) -> None:
         _, exc, _ = sys.exc_info()
 
-        # Warn when a logger is used outside of a run context, the stack level here
-        # gets us to the user logging call
         if isinstance(exc, MissingContextError):
-            warnings.warn(exc, stacklevel=8)
-            return
+            log_handling_when_missing_flow = (
+                PREFECT_LOGGING_ORION_WHEN_MISSING_FLOW.value()
+            )
+            if log_handling_when_missing_flow == "warn":
+                # Warn when a logger is used outside of a run context, the stack level here
+                # gets us to the user logging call
+                warnings.warn(exc, stacklevel=8)
+                return
+            elif log_handling_when_missing_flow == "ignore":
+                return
+            else:
+                raise exc
 
         # Display a longer traceback for other errors
         return super().handleError(record)
diff --git a/src/prefect/settings.py b/src/prefect/settings.py
--- a/src/prefect/settings.py
+++ b/src/prefect/settings.py
@@ -64,6 +64,7 @@
 import pydantic
 import toml
 from pydantic import BaseSettings, Field, create_model, root_validator, validator
+from typing_extensions import Literal
 
 from prefect.exceptions import MissingProfileError
 from prefect.utilities.names import OBFUSCATED_PREFIX, obfuscate
@@ -583,6 +584,20 @@ def default_cloud_ui_url(settings, value):
 )
 """The maximum size in bytes for a single log."""
 
+PREFECT_LOGGING_ORION_WHEN_MISSING_FLOW = Setting(
+    Literal["warn", "error", "ignore"],
+    default="warn",
+)
+"""
+Controls the behavior when loggers attempt to send logs to Orion without a flow run id.
+The Orion log handler can only send logs within flow run contexts unless the flow run id is
+manually provided.
+
+"warn": Log a warning message.
+"error": Raise an error.
+"ignore": Do not log a warning message or raise an error.
+"""
+
 PREFECT_LOGGING_COLORS = Setting(
     bool,
     default=True,


Test patch:
diff --git a/tests/test_logging.py b/tests/test_logging.py
--- a/tests/test_logging.py
+++ b/tests/test_logging.py
@@ -52,6 +52,7 @@
     PREFECT_LOGGING_ORION_BATCH_SIZE,
     PREFECT_LOGGING_ORION_ENABLED,
     PREFECT_LOGGING_ORION_MAX_LOG_SIZE,
+    PREFECT_LOGGING_ORION_WHEN_MISSING_FLOW,
     PREFECT_LOGGING_SETTINGS_PATH,
     temporary_settings,
 )
@@ -490,7 +491,7 @@ def test_does_not_send_logs_when_handler_is_disabled(
 
         mock_log_worker().enqueue.assert_not_called()
 
-    def test_does_not_send_logs_outside_of_run_context(
+    def test_does_not_send_logs_outside_of_run_context_with_default_setting(
         self, logger, mock_log_worker, capsys
     ):
         # Warns in the main process
@@ -505,6 +506,56 @@ def test_does_not_send_logs_outside_of_run_context(
         output = capsys.readouterr()
         assert output.err == ""
 
+    def test_does_not_send_logs_outside_of_run_context_with_error_setting(
+        self, logger, mock_log_worker, capsys
+    ):
+        with temporary_settings(
+            updates={PREFECT_LOGGING_ORION_WHEN_MISSING_FLOW: "error"},
+        ):
+            with pytest.raises(
+                MissingContextError,
+                match="attempted to send logs .* without a flow run id",
+            ):
+                logger.info("test")
+
+        mock_log_worker().enqueue.assert_not_called()
+
+        # No stderr output
+        output = capsys.readouterr()
+        assert output.err == ""
+
+    def test_does_not_send_logs_outside_of_run_context_with_ignore_setting(
+        self, logger, mock_log_worker, capsys
+    ):
+        with temporary_settings(
+            updates={PREFECT_LOGGING_ORION_WHEN_MISSING_FLOW: "ignore"},
+        ):
+            logger.info("test")
+
+        mock_log_worker().enqueue.assert_not_called()
+
+        # No stderr output
+        output = capsys.readouterr()
+        assert output.err == ""
+
+    def test_does_not_send_logs_outside_of_run_context_with_warn_setting(
+        self, logger, mock_log_worker, capsys
+    ):
+        with temporary_settings(
+            updates={PREFECT_LOGGING_ORION_WHEN_MISSING_FLOW: "warn"},
+        ):
+            # Warns in the main process
+            with pytest.warns(
+                UserWarning, match="attempted to send logs .* without a flow run id"
+            ):
+                logger.info("test")
+
+        mock_log_worker().enqueue.assert_not_called()
+
+        # No stderr output
+        output = capsys.readouterr()
+        assert output.err == ""
+
     def test_missing_context_warning_refers_to_caller_lineno(
         self, logger, mock_log_worker
     ):
@@ -517,7 +568,7 @@ def test_missing_context_warning_refers_to_caller_lineno(
             logger.info("test")
             lineno = getframeinfo(currentframe()).lineno - 1
             # The above dynamic collects the line number so that added tests do not
-            # break this tests
+            # break this test
 
         mock_log_worker().enqueue.assert_not_called()
         assert warnings.pop().lineno == lineno