Repository: DataDog/dd-trace-py

Issue:
Internal logger fails until logging is patched because dd.service is not set
### Which version of dd-trace-py are you using?
0.46.0

### Which version of the libraries are you using?
datadog 0.39.0
ddtrace 0.46.0

### How can we reproduce your problem?

1. Make a toy project that uses `jinja2`, or some other module from `contrib/`, and `logging`.
2. Enable these env vars:
```
DD_SERVICE=foo
DD_TRACE_DEBUG=true
DD_TRACE_STARTUP_LOGS=true
```
3. Start the project with `ddtrace-run` and observe that the internal logger fails until `logging` is patched.

### What is the result that you get?

Every time the `ddtrace` internal logger tries to log something, it raises an exception like this:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/logging/__init__.py", line 430, in format
    return self._format(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 426, in _format
    return self._fmt % record.__dict__
KeyError: 'dd.service'
member_backend              |
During handling of the above exception, another exception occurred:
member_backend              |
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1079, in emit
    msg = self.format(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 923, in format
    return fmt.format(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 662, in format
    s = self.formatMessage(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 631, in formatMessage
    return self._style.format(record)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 432, in format
    raise ValueError('Formatting field not found in record: %s' % e)
ValueError: Formatting field not found in record: 'dd.service'
Call stack:
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site.py", line 589, in <module>
    main()
  File "/usr/local/lib/python3.9/site.py", line 582, in main
    execsitecustomize()
  File "/usr/local/lib/python3.9/site.py", line 521, in execsitecustomize
    import sitecustomize
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/ddtrace/bootstrap/sitecustomize.py", line 17, in <module>
    from ddtrace.utils.formats import asbool, get_env, parse_tags_str  # noqa
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 972, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/ddtrace/__init__.py", line 18, in <module>
    tracer = Tracer()
  File "/usr/local/lib/python3.9/site-packages/ddtrace/tracer.py", line 146, in __init__
    self.configure(
  File "/usr/local/lib/python3.9/site-packages/ddtrace/vendor/debtcollector/removals.py", line 261, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/ddtrace/vendor/debtcollector/removals.py", line 261, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/ddtrace/tracer.py", line 362, in configure
    self._log_compat(logging.WARNING, msg)
  File "/usr/local/lib/python3.9/site-packages/ddtrace/tracer.py", line 590, in _log_compat
    self.log.log(level, msg)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1500, in log
    self._log(level, msg, args, **kwargs)
  File "/usr/local/lib/python3.9/logging/__init__.py", line 1577, in _log
    self.handle(record)
  File "/usr/local/lib/python3.9/site-packages/ddtrace/internal/logger.py", line 118, in handle
    super(DDLogger, self).handle(record)
Message: '- DATADOG TRACER DIAGNOSTIC - Agent not reachable at datadog:8126. Exception raised: [Errno 111] Connection refused'
Arguments: ()
```
I've confirmed that `$DD_SERVICE` is set before `ddtrace-run` is called.

When it finally patches `logging`, all the exceptions go away.

```
2021-01-26 19:53:06,768 INFO [ddtrace.monkey] [monkey.py:171] [dd.service=backend dd.env=dev dd.version= dd.trace_id=0 dd.span_id=0] - patched 1/1 modules (logging)
```
Notice that `dd.service` is now set.

I think this is a bug in `ddtrace`. It needs to patch `logging` first so that its own internal logger will work.

### What is the result that you expected?

There shouldn't be any exceptions when `ddtrace` starts up.


Discussion:
Hey @ryboe 

Have you found any solution for this?
No, sorry. I haven't looked at this problem in months.
Thanks! Seems like it is more pain to use APM than the gain it provides. 
Having this exact issue fwiw, with `ddtrace` 0.51.2.
Having the same issue with `ddtrace = 0.53.0`
Sorry everyone for the log delay and not acknowledging until now.

I have been able to reproduce this easily by running `DD_LOGS_INJECTION=true DD_TRACE_DEBUG=true ddtrace-run python -c ''`.

I have opened a PR #2905 to address this issue.

The problem occurs because the tracer is initialized on import of `ddtrace`, will initialize logging (`logging.basicConfig`), and then will log as part of initialization of the tracer... before we get a chance to patch logging.

We fixed this issue in our `sitecustomize.py`, but that isn't good enough.

Resolving patch:
diff --git a/ddtrace/contrib/logging/patch.py b/ddtrace/contrib/logging/patch.py
--- a/ddtrace/contrib/logging/patch.py
+++ b/ddtrace/contrib/logging/patch.py
@@ -39,7 +39,9 @@ def _get_current_span(tracer=None):
     if not tracer:
         tracer = ddtrace.tracer
 
-    if not tracer.enabled:
+    # We might be calling this during library initialization, in which case `ddtrace.tracer` might
+    # be the `tracer` module and not the global tracer instance.
+    if not getattr(tracer, "enabled", False):
         return None
 
     return tracer.current_span()
diff --git a/ddtrace/tracer.py b/ddtrace/tracer.py
--- a/ddtrace/tracer.py
+++ b/ddtrace/tracer.py
@@ -49,6 +49,7 @@
 from .internal.writer import AgentWriter
 from .internal.writer import LogWriter
 from .internal.writer import TraceWriter
+from .monkey import patch
 from .provider import DefaultContextProvider
 from .sampler import BasePrioritySampler
 from .sampler import BaseSampler
@@ -72,6 +73,8 @@
 )
 if debug_mode and not hasHandlers(log) and call_basic_config:
     if config.logs_injection:
+        # We need to ensure logging is patched in case the tracer logs during initialization
+        patch(logging=True)
         logging.basicConfig(level=logging.DEBUG, format=DD_LOG_FORMAT)
     else:
         logging.basicConfig(level=logging.DEBUG)


Test patch:
diff --git a/tests/integration/test_integration.py b/tests/integration/test_integration.py
--- a/tests/integration/test_integration.py
+++ b/tests/integration/test_integration.py
@@ -601,3 +601,33 @@ def test_partial_flush_log(run_python_code_in_subprocess):
     log.debug.assert_has_calls(calls)
     s1.finish()
     t.shutdown()
+
+
+def test_ddtrace_run_startup_logging_injection(ddtrace_run_python_code_in_subprocess):
+    """
+    Regression test for enabling debug logging and logs injection
+
+    When both DD_TRACE_DEBUG and DD_LOGS_INJECTION are enabled
+    any logging during tracer initialization would raise an exception
+    because `dd.service` was not available in the log record yet.
+    """
+    env = os.environ.copy()
+    env["DD_TRACE_DEBUG"] = "true"
+    env["DD_LOGS_INJECTION"] = "true"
+
+    # DEV: We don't actually have to execute any code to validate this
+    out, err, status, pid = ddtrace_run_python_code_in_subprocess("", env=env)
+
+    # The program will always exit successfully
+    # Errors during logging do not crash the app
+    assert status == 0, (out, err)
+
+    # The program does nothing
+    assert out == b""
+
+    # stderr is expected to log something due to debug logging
+    assert b"[dd.service= dd.env= dd.version= dd.trace_id=0 dd.span_id=0]" in err
+
+    # Assert no logging exceptions in stderr
+    assert b"KeyError: 'dd.service'" not in err
+    assert b"ValueError: Formatting field not found in record: 'dd.service'" not in err