Repository: mlflow/mlflow

Issue:
[BUG] Logging config is difficult to override
### Issues Policy acknowledgement

- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)

### Where did you encounter this bug?

Local machine

### Willingness to contribute

Yes. I can contribute a fix for this bug independently.

### MLflow version

- Client: 2.10.2/master
- Tracking server: 2.10.2/master


### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: all
- **Python version**: 3.12
- **yarn version, if running the dev UI**: n/a


### Describe the problem

Mlflow [configures logging in the `__init__.py`](https://github.com/mlflow/mlflow/blob/a030e40a4b61608b5982c2b515e60916f6299a9e/mlflow/__init__.py#L91). I understand this was added in https://github.com/mlflow/mlflow/commit/28627903b22c8d365295d6057be392ab7b03a6a2 to provide more info i.e. timestamps as part of #723.

Mlflow's logging config might not be what a user wants and is difficult to override since logging config is global in python. 

This can be especially problematic when running an mlflow in an environment that send logs to a log aggregator which expects them in a specific format like json. The mlflow logs can slip through as formatted in `_configure_logging` or may be duplicated if additional handlers are added without removing the mlflow handler.

I propose either adding an environment variable that keeps the existing behavior or removing the logging config in the `__init__.py`. I will submit a PR with the environment variable.

Example of attempting to remove the logging config. This mostly works, so long as one does this **after** importing mlflow. It's quite verbose and can be brittle.

```python
# Wipe out the logging config entirely
CFG = {"version": 1, "formatters": {}, "handlers": {}, "loggers": {}}
logging.root.handlers.clear()
logging.config.dictConfig(CFG)

# In case anything survived that... which the mlflow loggers can
# Clear off all existing handlers from the root
while logging.root.hasHandlers():
    logging.root.removeHandler(logging.root.handlers[0])

# Remove handlers and reset the level for deps that manually set them up
# i.e. mlflow stream handler. These aren't cleared by root.
loggers = logging.Logger.manager.loggerDict.keys()
for logger_name in loggers:
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.WARNING)
    # Need to make sure to propagate back up the hierarchy or they
    # will use a default handler instead of root's handler, causing logs to not be formatted if root should be formatting them
    logger.propagate = True

    # Avoid iterating a list while modifying it
    while len(logger.handlers) > 0:
        logger.removeHandler(logger.handlers[0])

# Now you can do a logging config
logging.config.dictConfig({...})

```



### Tracking information

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
```


### Code to reproduce issue

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
import mlflow
# do custom logging config
...

# This logger will escape the logging config unless you explicitly remove its handlers
logging.getLogger("mlflow').info("hello")
```


### Stack trace

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
REPLACE_ME
```


### Other info / logs

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
# Actual
2018/11/20 13:11:28 WARNING mlflow.server mlflow error
# multi-line traceback here instead of a json log
Traceback (most recent call last):
...

# Desired
{"event": "mlflow error", "level": "warning", "logger": "mlflow.server", "timestamp": "2024-01-22T15:20:17.075588Z", "exception": "Traceback (most recent call last):\n  ....", "func_name": "the_function", "filename": "the_file.py", "lineno": 94}```
```

### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/deployments`: MLflow Deployments client APIs, server, and third-party Deployments integrations
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [X] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations
[BUG] Logging config is difficult to override
### Issues Policy acknowledgement

- [X] I have read and agree to submit bug reports in accordance with the [issues policy](https://www.github.com/mlflow/mlflow/blob/master/ISSUE_POLICY.md)

### Where did you encounter this bug?

Local machine

### Willingness to contribute

Yes. I can contribute a fix for this bug independently.

### MLflow version

- Client: 2.10.2/master
- Tracking server: 2.10.2/master


### System information

- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**: all
- **Python version**: 3.12
- **yarn version, if running the dev UI**: n/a


### Describe the problem

Mlflow [configures logging in the `__init__.py`](https://github.com/mlflow/mlflow/blob/a030e40a4b61608b5982c2b515e60916f6299a9e/mlflow/__init__.py#L91). I understand this was added in https://github.com/mlflow/mlflow/commit/28627903b22c8d365295d6057be392ab7b03a6a2 to provide more info i.e. timestamps as part of #723.

Mlflow's logging config might not be what a user wants and is difficult to override since logging config is global in python. 

This can be especially problematic when running an mlflow in an environment that send logs to a log aggregator which expects them in a specific format like json. The mlflow logs can slip through as formatted in `_configure_logging` or may be duplicated if additional handlers are added without removing the mlflow handler.

I propose either adding an environment variable that keeps the existing behavior or removing the logging config in the `__init__.py`. I will submit a PR with the environment variable.

Example of attempting to remove the logging config. This mostly works, so long as one does this **after** importing mlflow. It's quite verbose and can be brittle.

```python
# Wipe out the logging config entirely
CFG = {"version": 1, "formatters": {}, "handlers": {}, "loggers": {}}
logging.root.handlers.clear()
logging.config.dictConfig(CFG)

# In case anything survived that... which the mlflow loggers can
# Clear off all existing handlers from the root
while logging.root.hasHandlers():
    logging.root.removeHandler(logging.root.handlers[0])

# Remove handlers and reset the level for deps that manually set them up
# i.e. mlflow stream handler. These aren't cleared by root.
loggers = logging.Logger.manager.loggerDict.keys()
for logger_name in loggers:
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.WARNING)
    # Need to make sure to propagate back up the hierarchy or they
    # will use a default handler instead of root's handler, causing logs to not be formatted if root should be formatting them
    logger.propagate = True

    # Avoid iterating a list while modifying it
    while len(logger.handlers) > 0:
        logger.removeHandler(logger.handlers[0])

# Now you can do a logging config
logging.config.dictConfig({...})

```



### Tracking information

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
```


### Code to reproduce issue

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
import mlflow
# do custom logging config
...

# This logger will escape the logging config unless you explicitly remove its handlers
logging.getLogger("mlflow').info("hello")
```


### Stack trace

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
REPLACE_ME
```


### Other info / logs

<!-- PLEASE KEEP BACKTICKS AND CHECK PREVIEW -->
```
# Actual
2018/11/20 13:11:28 WARNING mlflow.server mlflow error
# multi-line traceback here instead of a json log
Traceback (most recent call last):
...

# Desired
{"event": "mlflow error", "level": "warning", "logger": "mlflow.server", "timestamp": "2024-01-22T15:20:17.075588Z", "exception": "Traceback (most recent call last):\n  ....", "func_name": "the_function", "filename": "the_file.py", "lineno": 94}```
```

### What component(s) does this bug affect?

- [ ] `area/artifacts`: Artifact stores and artifact logging
- [ ] `area/build`: Build and test infrastructure for MLflow
- [ ] `area/deployments`: MLflow Deployments client APIs, server, and third-party Deployments integrations
- [ ] `area/docs`: MLflow documentation pages
- [ ] `area/examples`: Example code
- [ ] `area/model-registry`: Model Registry service, APIs, and the fluent client calls for Model Registry
- [ ] `area/models`: MLmodel format, model serialization/deserialization, flavors
- [ ] `area/recipes`: Recipes, Recipe APIs, Recipe configs, Recipe Templates
- [ ] `area/projects`: MLproject format, project running backends
- [ ] `area/scoring`: MLflow Model server, model deployment tools, Spark UDFs
- [ ] `area/server-infra`: MLflow Tracking server backend
- [X] `area/tracking`: Tracking Service, tracking client APIs, autologging

### What interface(s) does this bug affect?

- [ ] `area/uiux`: Front-end, user experience, plotting, JavaScript, JavaScript dev server
- [ ] `area/docker`: Docker use across MLflow's components, such as MLflow Projects and MLflow Models
- [ ] `area/sqlalchemy`: Use of SQLAlchemy in the Tracking Service or Model Registry
- [ ] `area/windows`: Windows support

### What language(s) does this bug affect?

- [ ] `language/r`: R APIs and clients
- [ ] `language/java`: Java APIs and clients
- [ ] `language/new`: Proposals for new client languages

### What integration(s) does this bug affect?

- [ ] `integrations/azure`: Azure and Azure ML integrations
- [ ] `integrations/sagemaker`: SageMaker integrations
- [ ] `integrations/databricks`: Databricks integrations


Discussion:

Resolving patch:
diff --git a/mlflow/__init__.py b/mlflow/__init__.py
--- a/mlflow/__init__.py
+++ b/mlflow/__init__.py
@@ -32,7 +32,6 @@
 from mlflow.version import VERSION
 
 __version__ = VERSION
-
 from mlflow import (
     artifacts,  # noqa: F401
     client,  # noqa: F401
@@ -42,6 +41,7 @@
     projects,  # noqa: F401
     tracking,  # noqa: F401
 )
+from mlflow.environment_variables import MLFLOW_CONFIGURE_LOGGING
 from mlflow.utils.lazy_load import LazyLoader
 from mlflow.utils.logging_utils import _configure_mlflow_loggers
 
@@ -82,7 +82,8 @@
 transformers = LazyLoader("mlflow.transformers", globals(), "mlflow.transformers")
 xgboost = LazyLoader("mlflow.xgboost", globals(), "mlflow.xgboost")
 
-_configure_mlflow_loggers(root_module_name=__name__)
+if MLFLOW_CONFIGURE_LOGGING.get() is True:
+    _configure_mlflow_loggers(root_module_name=__name__)
 
 from mlflow.client import MlflowClient
 from mlflow.exceptions import MlflowException
diff --git a/mlflow/environment_variables.py b/mlflow/environment_variables.py
--- a/mlflow/environment_variables.py
+++ b/mlflow/environment_variables.py
@@ -511,3 +511,9 @@ def get(self):
 MLFLOW_GATEWAY_RATE_LIMITS_STORAGE_URI = _EnvironmentVariable(
     "MLFLOW_GATEWAY_RATE_LIMITS_STORAGE_URI", str, None
 )
+
+#: Specifies whether or not to have mlflow configure logging on import.
+#: If set to True, mlflow will configure ``mlflow.<module_name>`` loggers with
+#: logging handlers and formatters.
+#: (default: ``True``)
+MLFLOW_CONFIGURE_LOGGING = _BooleanEnvironmentVariable("MLFLOW_LOGGING_CONFIGURE_LOGGING", True)


Test patch: