Repository: iterative/dvc

Issue:
`dvc.cli.main` disables logging of other packages
## Description
I'm developing the https://github.com/zincware/ZnTrack package to interface with DVC.
I was relying on `subprocess.check_call(["dvc", ...])` but wanted to switch to `dvc.cli.main([...])` instead.
Using the DVC API gives a noticeable performance increase.
Unfortunately, doing so the logger of my package will be permanently disabled.

### Reproduce

```python
import logging
import sys
import dvc.cli

logger = logging.getLogger("mypackage")
logger.setLevel(logging.DEBUG)

channel = logging.StreamHandler(sys.stdout)
channel.setLevel(logging.DEBUG)

logger.addHandler(channel)

print(logger.disabled)
# False
dvc.cli.main(["init"])
print(logger.disabled)
# True
```

### Expected

I know the `dvc.cli` API is currently not exposed but it would be helpful if it would not disable logging of other python packages.

### Fix

capture the  disabled loggers from
https://github.com/iterative/dvc/blob/bb80eab77ae5830790113acd1b5b49bf311e5127/dvc/cli/__init__.py#L159
and enable them at the end.
If this would be an accepted fix, I can implement it and make a PR?





Discussion:
> I was relying on subprocess.check_call(["dvc", ...]) but wanted to switch to dvc.cli.main([...]) instead.
Using the DVC API gives a noticeable performance increase.

hi @PythonFZ , don't mean to dismiss this request, but have you considered using the [ `dvc.repo.Repo` ](https://github.com/iterative/dvc/blob/main/dvc/repo/__init__.py#L53)API? 

It should be more efficient than `cli.main` calls while still providing the same (minus some UI) features. For example, you could reuse the same `Repo` instance for multiple calls.
That is what I want to do eventually.
I'm already using it for my DVC and HPC package https://github.com/zincware/dask4dvc but I'm still in the progress of understanding. Switching to cli already sped up my Test suit by a Factor of 3 and I was able to implement it in a few minuts ( minus testing)

We might be able to get rid of `disable_other_loggers()` in `main()`, since we lazily import most of the libraries, so it's probably not doing anything at all. I see it disabling the following loggers:
```
tqdm.cli
tqdm
asyncio
aiohttp
dvc_objects
dvc_data
```

`tqdm.cli` and `tqdm` should not be a problem, `asyncio` and `aiohttp` are already explicitly changed to `CRITICAL` only in `dvc.logger.setup()` and the last two are explicitly enabled later through config. Even then, we can explicitly disable them if we want to.

I am a bit hesitant to add any arguments to `main()`, because as it exists, it is meant to be an entrypoint to dvc cli. (Although in the future, I am open to introduce `console_main()`/`cli_main()` for cli and `main()` for programmatic access).
Note that DVC only has one stable API: the command line. The `dvc.cli.main` also is guaranteed to be backward-compatible as it is an entry point to the commandline even if it's undocumented. `Repo` api has no such guarantees, although they don't change very often.
The following should work as a replacement for the `disable_other_loggers()` increasing the execution time on my machine  from `6.14 µs ± 126 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)` to `7.31 µs ± 613 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each)`.  Changing `if logger_name != "dvc" and not logger_name.startswith("dvc.")` to `if not logger_name.startswith("dvc")` gave a performance incrase of about `1 µs`.

Would this be an acceptable solution to you?

```python
import contextlib

@contextlib.contextmanager
def disable_other_loggers():
    logging.captureWarnings(True)
    loggerDict = logging.root.manager.loggerDict  # pylint: disable=no-member

    disabled_logger_list = []

    for logger_name, logger in loggerDict.items():
        if not logger_name.startswith("dvc"): 
            if not logger.disabled:
                disabled_logger_list.append(logger)
                logger.disabled = True
    try:
        yield
    finally:
        for logger in disabled_logger_list:
            logger.disabled = False
```
and then

```python
with disable_other_loggers():
   ...
```

Resolving patch:
diff --git a/dvc/cli/__init__.py b/dvc/cli/__init__.py
--- a/dvc/cli/__init__.py
+++ b/dvc/cli/__init__.py
@@ -153,7 +153,7 @@ def main(argv=None):  # noqa: C901
     from dvc._debug import debugtools
     from dvc.config import ConfigError
     from dvc.exceptions import DvcException, NotDvcRepoError
-    from dvc.logger import disable_other_loggers, set_loggers_level
+    from dvc.logger import set_loggers_level
 
     # NOTE: stderr/stdout may be closed if we are running from dvc.daemon.
     # On Linux we directly call cli.main after double forking and closing
@@ -166,7 +166,6 @@ def main(argv=None):  # noqa: C901
         logging.disable(logging.INFO)
 
     args = None
-    disable_other_loggers()
 
     outerLogLevel = logger.level
     try:
diff --git a/dvc/logger.py b/dvc/logger.py
--- a/dvc/logger.py
+++ b/dvc/logger.py
@@ -180,14 +180,6 @@ def _iter_causes(exc):
         exc = exc.__cause__
 
 
-def disable_other_loggers():
-    logging.captureWarnings(True)
-    loggerDict = logging.root.manager.loggerDict  # pylint: disable=no-member
-    for logger_name, logger in loggerDict.items():
-        if logger_name != "dvc" and not logger_name.startswith("dvc."):
-            logger.disabled = True  # type: ignore[union-attr]
-
-
 def set_loggers_level(level: int = logging.INFO) -> None:
     for name in ["dvc", "dvc_objects", "dvc_data"]:
         logging.getLogger(name).setLevel(level)
diff --git a/dvc/repo/experiments/executor/base.py b/dvc/repo/experiments/executor/base.py
--- a/dvc/repo/experiments/executor/base.py
+++ b/dvc/repo/experiments/executor/base.py
@@ -716,13 +716,10 @@ def _raise_ref_conflict(scm, ref, new_rev, checkpoint):
 
     @staticmethod
     def _set_log_level(level):
-        from dvc.logger import disable_other_loggers
-
         # When executor.reproduce is run in a multiprocessing child process,
         # dvc.cli.main will not be called for that child process so we need to
         # setup logging ourselves
         dvc_logger = logging.getLogger("dvc")
-        disable_other_loggers()
         if level is not None:
             dvc_logger.setLevel(level)
 


Test patch:
diff --git a/tests/dir_helpers.py b/tests/dir_helpers.py
--- a/tests/dir_helpers.py
+++ b/tests/dir_helpers.py
@@ -49,8 +49,6 @@
 
 import pytest
 
-from dvc.logger import disable_other_loggers
-
 __all__ = [
     "run_copy",
     "run_head",
@@ -60,9 +58,6 @@
     "git_downstream",
 ]
 
-# see https://github.com/iterative/dvc/issues/3167
-disable_other_loggers()
-
 
 @pytest.fixture
 def run_copy(tmp_dir, copy_script, dvc):