Repository: open-telemetry/opentelemetry-python

Issue:
opentelemetry-instrument and logs SDK cause logs to be swallowed by default
Linux, python3.9

**Steps to reproduce**
In a fresh venv
```py
pip install opentelemetry-distro opentelemetry-instrumentation
opentelemetry-instrument python -c 'import logging; logger = logging.getLogger("foo"); logger.error("oh no")'
```

**What is the expected behavior?**
See the error printed to the console

**What is the actual behavior?**
No output

**Additional context**
I believe the LogHandler is being installed on the root logger, but there is no log processor/exporter. Which causes logging to be no-op!


Discussion:
@aabmass You are using the `logging.error` instead of `logger.error`. Can you confirm "oh no" printed to console if `logger` is used? Still a bug though.
Oops. Yes it's the same behavior. Updated the issue description
I think I know what's happening. When we log a message, `logging.basicConfig` will be called and `StreamHandler` is added. But the `basicConfig` does nothing if there are some handlers already configured. Since we are adding `OTELHandler` during auto-instrumentation pipeline early any later call to `basicConfig` is ineffective and `StreamHandler` is never appended (unless someone calls `basicConfig` with force=True) Which is why error is not printed to console. 
catch-22 situation - If we add `logging.basicConfig()` call [here](https://github.com/open-telemetry/opentelemetry-python/blob/main/opentelemetry-sdk/src/opentelemetry/sdk/_configuration/__init__.py#L118-L119) this issue should get fixed but can be surprising for users who might have their custom `logging.basicConfig(config)` in application code. If we don't then the current issue exists and can be problematic. Any ideas?
I can't think of any better solution than disabling the logging auto instrumentation. Does anyone have other ideas?
Please do so and let me get some good sleep... Apologies for being unprofessional.
> catch-22 situation - If we add `logging.basicConfig()` call [here](https://github.com/open-telemetry/opentelemetry-python/blob/main/opentelemetry-sdk/src/opentelemetry/sdk/_configuration/__init__.py#L118-L119) this issue should get fixed but can be surprising for users who might have their custom `logging.basicConfig(config)` in application code.

@srikanthccv I think it's reasonable to expect autoinstrumentation to mess with existing logging pipelines. If they want to override, they can do `force=True` like you mentioned. OTOH, I think printing logs to stdout should still happen by default.

Maybe the least surprising behavior would be to have our log handler off by default, and only turn it on if an logs exporter is specifically configured (or have another flag). How important is it for first time autoinstrumentation users to have logs automagically sent to the collector?
> Maybe the least surprising behavior would be to have our log handler off by default, and only turn it on if an logs exporter is specifically configured (or have another flag).

Yes, we discussed this in today's meeting. We agreed on disabling the logging pipeline setup by default but introduce a flag or env which can be used to enable this with clear documentation until we find a clean solution. I think some people have applications where `basicConfig` is invoked prior to our `_init_logging` setup call and it's working fine for them. 

> How important is it for first time autoinstrumentation users to have logs automagically sent to the collector?

Not sure about the importance. We are doing this out of the box automatic pipeline setup for each signal as soon as they have good SDK support.
Great thank you! Let's leave this issue open after the temp fix is made to see if someone has a better idea.

As a crazy idea, we could patch `logging.basicConfig` in the autoinstrumentation agent to make it do the behavior we want.
The basic log initialization can be achieved within the instrumentation phase, not the configuration phase, if the following criteria are met:

1. A certain kinds of instrumentations can have precedence on application order over other instrumentators.  This guarantees that the logging instrumentation come first in the order of application. Some instrumentators should need to get the instrumentee initialized before applying instrumentation and this may induce the call to `logging.basicConfig()` at an unwanted timing.
2. It is assured that no code invokes the calls that involves the root logger, specifically `logging.{exception,critical,error,warning,info,debug,log}()`, `logging.getLogger(None)`, and `logging.basicConfig()` aside from `logging.shutdown()`.  The only allowed call to the standard `logging` package has to be `logging.getLogger(...)` with some non-None argument.  It should be better of having a logger of some name, as a utility logger that is exposed to the instrumentators within the public API, that has a stream handler to stderr attached.


Resolving patch:
diff --git a/opentelemetry-sdk/src/opentelemetry/sdk/_configuration/__init__.py b/opentelemetry-sdk/src/opentelemetry/sdk/_configuration/__init__.py
--- a/opentelemetry-sdk/src/opentelemetry/sdk/_configuration/__init__.py
+++ b/opentelemetry-sdk/src/opentelemetry/sdk/_configuration/__init__.py
@@ -18,6 +18,7 @@
 """
 
 import logging
+import os
 from abc import ABC, abstractmethod
 from os import environ
 from typing import Dict, Optional, Sequence, Tuple, Type
@@ -36,6 +37,9 @@
     set_log_emitter_provider,
 )
 from opentelemetry.sdk._logs.export import BatchLogProcessor, LogExporter
+from opentelemetry.sdk.environment_variables import (
+    _OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED,
+)
 from opentelemetry.sdk.resources import Resource
 from opentelemetry.sdk.trace import TracerProvider
 from opentelemetry.sdk.trace.export import BatchSpanProcessor, SpanExporter
@@ -185,7 +189,11 @@ def _initialize_components(auto_instrumentation_version):
     id_generator_name = _get_id_generator()
     id_generator = _import_id_generator(id_generator_name)
     _init_tracing(trace_exporters, id_generator, auto_instrumentation_version)
-    _init_logging(log_exporters, auto_instrumentation_version)
+    logging_enabled = os.getenv(
+        _OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED, "false"
+    )
+    if logging_enabled.strip().lower() == "true":
+        _init_logging(log_exporters, auto_instrumentation_version)
 
 
 class _BaseConfigurator(ABC):
diff --git a/opentelemetry-sdk/src/opentelemetry/sdk/environment_variables.py b/opentelemetry-sdk/src/opentelemetry/sdk/environment_variables.py
--- a/opentelemetry-sdk/src/opentelemetry/sdk/environment_variables.py
+++ b/opentelemetry-sdk/src/opentelemetry/sdk/environment_variables.py
@@ -407,6 +407,20 @@
 LogEmitterProvider is used.
 """
 
+_OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED = (
+    "OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED"
+)
+"""
+.. envvar:: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
+
+The :envvar:`OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED` environment variable allows users to
+enable/disabe the logging SDK auto instrumentation.
+Default: False
+
+Note: Logs SDK and its related settings are experimental.
+"""
+
+
 OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE = (
     "OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE"
 )


Test patch:
diff --git a/opentelemetry-sdk/tests/test_configurator.py b/opentelemetry-sdk/tests/test_configurator.py
--- a/opentelemetry-sdk/tests/test_configurator.py
+++ b/opentelemetry-sdk/tests/test_configurator.py
@@ -29,6 +29,7 @@
     _import_id_generator,
     _init_logging,
     _init_tracing,
+    _initialize_components,
 )
 from opentelemetry.sdk._logs import LoggingHandler
 from opentelemetry.sdk._logs.export import ConsoleLogExporter
@@ -277,6 +278,31 @@ def test_logging_init_exporter(self):
         logging.getLogger(__name__).error("hello")
         self.assertTrue(provider.processor.exporter.export_called)
 
+    @patch.dict(
+        environ,
+        {"OTEL_RESOURCE_ATTRIBUTES": "service.name=otlp-service"},
+    )
+    @patch("opentelemetry.sdk._configuration._init_tracing")
+    @patch("opentelemetry.sdk._configuration._init_logging")
+    def test_logging_init_disable_default(self, logging_mock, tracing_mock):
+        _initialize_components("auto-version")
+        self.assertEqual(logging_mock.call_count, 0)
+        self.assertEqual(tracing_mock.call_count, 1)
+
+    @patch.dict(
+        environ,
+        {
+            "OTEL_RESOURCE_ATTRIBUTES": "service.name=otlp-service",
+            "OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED": "True",
+        },
+    )
+    @patch("opentelemetry.sdk._configuration._init_tracing")
+    @patch("opentelemetry.sdk._configuration._init_logging")
+    def test_logging_init_enable_env(self, logging_mock, tracing_mock):
+        _initialize_components("auto-version")
+        self.assertEqual(logging_mock.call_count, 1)
+        self.assertEqual(tracing_mock.call_count, 1)
+
 
 class TestExporterNames(TestCase):
     def test_otlp_exporter_overwrite(self):