Repository: pytorch/ignite

Issue:
Cannot set log level of `ignite.distributed` because of calls to setup_logger inside auto functions
## 🐛 Bug description

<!-- A clear and concise description of what the bug is. -->
<!-- Please, add steps on how to reproduce it. -->
<!-- If you have a code sample, error messages, stack traces, please provide it here as well -->
<!-- A clear and concise description of what you expected to happen. -->

The `ignite.distributed` auto functions such as `ignite.distributed.auto_dataloader` use `setup_logger` to get a `logging.Logger` instance. However, this function always sets the logging level to `logging.INFO` and furthermore removes ands sets up the logging handlers. Functions should not be setting up a logger object if it is already initialized such that user code can specify a different log level, handlers, or log message format string in their code (e.g., main function).

See for example:
- https://github.com/pytorch/ignite/blob/6d490c8f3083f21ba8f82eb646d9052a06fae3e2/ignite/distributed/auto.py#L67
- https://github.com/pytorch/ignite/blob/6d490c8f3083f21ba8f82eb646d9052a06fae3e2/ignite/distributed/launcher.py#L201

Note that in case of `ignite.distributed.launcher.Parallel` one can modify the `self.logger` instance, but this logger already prints messages right after `setup_logger` in its `__init__` function, hence, making it impossible to adjust the log level, format, or handlers for these first messages also.

## Environment

- PyTorch Version (e.g., 1.4):
- Ignite Version (e.g., 0.3.0):
- OS (e.g., Linux):
- How you installed Ignite (`conda`, `pip`, source):
- Python version:
- Any other relevant information:



Discussion:
Thanks for the report @schuhschuh ! 
Yes, this part of code should be definitetly improved and I think @sdesrozis also proposed to do that.

@schuhschuh if you would like help us with improving the code, please feel free to discuss it here and then send a draft PR. Thanks
We could add logger argument to functions. Or define a logger for the module `distributed`.

@vfdev-5 other ideas ?
Defining a logger for the module could be nice but can be tricky as it should be adapted and re-adapted to the dist config...
I would say you already have global logging objects for these functions anyway by the use of `logging.getLogger()` in `setup_logger()`. This is what makes it possible to adjust the logger without accessing a module level logger instance reference. Multiple calls to `auto_*()` functions will give you the same logger instance. The issue is just that this instance will also always reset the level, format, and handlers.

Would it work to just use a `setup_logger` variant (maybe a flag for this function) which instead of removing any previous handlers and setting up the logger retrieved by `logging.getLogger(name)` would simply check `logger.hasHandlers()` and just return the already set up logger object if `True` at [this line](https://github.com/pytorch/ignite/blob/6d490c8f3083f21ba8f82eb646d9052a06fae3e2/ignite/utils.py#L142)? One could still check and remove all handlers for `distributed_rank > 0`. But given that these loggers have a `NullHandler` set by any previous call to the `auto_*` functions, I don't think this would be necessary. Also, by only checking `hasHandlers()` and making any changes to the logger object only if `False`, you leave the user the option to setup any desired handlers even for loggers where the `distributed_rank` of the process is greater than zero.

For example:
```
def setup_logger(
    name: Optional[str] = None,
    level: int = logging.INFO,
    stream: Optional[TextIO] = None,
    format: str = "%(asctime)s %(name)s %(levelname)s: %(message)s",
    filepath: Optional[str] = None,
    distributed_rank: Optional[int] = None,
    reset: bool = True,
) -> logging.Logger:
    logger = logging.getLogger(name)
    if logger.hasHandlers() and not reset:
        return logger
    # [...]
```

EDIT: It may be desirable to have the handlers always be removed and replaced by a `NullHandler` if `distributed_rank > 0` regardless of the `reset` flag (to be used in `auto_*()` functions when calling `setup_logger()`), though. Because then a user can setup the handler for the named logger in the main process before any processes are spawned.
@schuhschuh yes, I agree that we could use `setup_logger` for setting up and retrieving existing logger. 

@sdesrozis what do you think ?
Sorry for the lag, I missed this message !!

Just a remark. If user first calls `logging.basicConfig()` (or any configuration), the `root` logger is defined and embeds handlers. So after that, calling `setup_logger()` considering the suggestion of @schuhschuh should return the `root` logger initialized by the user... and it does not take into account the specific ignite handlers pushed in the method.

EDIT : This is the EDIT part of the last message of @schuhschuh

I'll look into it on my side.
Another remark. 

We use native `logging` module in `Engine` and `TerminateOnNan` and `LRFinder` (grep told me).

```python
logging.basicConfig(level=logging.INFO)

engine = Engine(lambda e, b: None)
engine.run([0, 1, 2], max_epochs=2)
```
prints in console
```
INFO:ignite.engine.engine.Engine:Engine run starting with max_epochs=2.
INFO:ignite.engine.engine.Engine:Epoch[1] Complete. Time taken: 00:00:00
INFO:ignite.engine.engine.Engine:Epoch[2] Complete. Time taken: 00:00:00
INFO:ignite.engine.engine.Engine:Engine run complete. Time taken: 00:00:00
```

Do we want that behavior ? IMO it would be better to use `setup_logger` in our modules rather than native `logging` as we have specific handlers for `distributed` mode. However, default level of `setup_logger` is `logging.INFO`. We would be better to have logger configured using `logging.ERROR` for internal.

What do you think ?
  
@sdesrozis Actually, setup_logger is using the native logging module. You are not actually retrieving the root logger, because getLogger is called with the module name as argument.The question should only be whether you are able to tell if the specific logger that you obtain, which is not a reference to the root logger, has had handlers set? I would think/hope so.
I agree that we always use native logging. I was talking about configuration, sorry it was not understandable.

If you call `setup_logger()` without argument, you get the root logger. I think that is a point to have in mind. `logging.basicConfig(...)` should not conflict with our way to setup loggers. If a logger is setup  with a specific name, I assume that it won't conflict. Finally I suppose it could help if `setup_logger()` returns a logger that differs from root.

What I said may be beside the point of the issue. I understand that multiple calls reset handlers and level. The suggested solution aims to configure once each logger. (Loggers must not have handlers attached, take care of root).

I will have a look on my own.

Resolving patch:
diff --git a/ignite/distributed/launcher.py b/ignite/distributed/launcher.py
--- a/ignite/distributed/launcher.py
+++ b/ignite/distributed/launcher.py
@@ -229,8 +229,6 @@ def __init__(
         self.backend = backend
         self._spawn_params = None
         self.init_method = init_method
-        self.logger = setup_logger(__name__ + "." + self.__class__.__name__, distributed_rank=0)
-        # distributed_rank=0 <=> explicit rank 0, avoid call idist. Critical for TPU on Colab, avoid context setup
 
         if self.backend is not None:
             if nproc_per_node is not None:
@@ -238,10 +236,8 @@ def __init__(
                     nproc_per_node, nnodes, node_rank, master_addr, master_port, init_method, **spawn_kwargs
                 )
 
-        if self._spawn_params is not None:
-            self.logger.info(f"Initialized distributed launcher with backend: '{self.backend}'")
-            msg = "\n\t".join([f"{k}: {v}" for k, v in self._spawn_params.items() if v is not None])
-            self.logger.info(f"- Parameters to spawn processes: \n\t{msg}")
+        # The logger will be setup after the idist.initialize() call
+        self._logger = None
 
     @staticmethod
     def _setup_spawn_params(
@@ -305,24 +301,43 @@ def training(local_rank, config, **kwargs):
 
         """
         if self._spawn_params is not None and self.backend is not None:
-            self.logger.info(f"Spawn function '{func}' in {self._spawn_params['nproc_per_node']} processes")
+            self._logger.info(  # type: ignore[attr-defined]
+                f"Spawn function '{func}' in {self._spawn_params['nproc_per_node']} processes"
+            )
             idist.spawn(self.backend, func, args=args, kwargs_dict=kwargs, **self._spawn_params)
         else:
-            self.logger.info(f"- Run '{func}' in {idist.get_world_size()} processes")
+            self._logger.info(f"- Run '{func}' in {idist.get_world_size()} processes")  # type: ignore[attr-defined]
             local_rank = idist.get_local_rank()
             func(local_rank, *args, **kwargs)
 
-        self.logger.info("End of run")
+        self._logger.info("End of run")  # type: ignore[attr-defined]
 
     def __enter__(self) -> "Parallel":
-        if (self.backend is not None) and self._spawn_params is None:
+        if self.backend is not None and self._spawn_params is None:
             idist.initialize(self.backend, init_method=self.init_method)
-            self.logger = setup_logger(__name__ + "." + self.__class__.__name__)
-            self.logger.info(f"Initialized processing group with backend: '{self.backend}'")
+
+        # The logger can be setup from now since idist.initialize() has been called (if needed)
+        self._logger = setup_logger(__name__ + "." + self.__class__.__name__)  # type: ignore[assignment]
+
+        if self.backend is not None:
+            if self._spawn_params is None:
+                self._logger.info(  # type: ignore[attr-defined]
+                    f"Initialized processing group with backend: '{self.backend}'"
+                )
+            else:
+                self._logger.info(  # type: ignore[attr-defined]
+                    f"Initialized distributed launcher with backend: '{self.backend}'"
+                )
+                msg = "\n\t".join([f"{k}: {v}" for k, v in self._spawn_params.items() if v is not None])
+                self._logger.info(  # type: ignore[attr-defined]
+                    f"- Parameters to spawn processes: \n\t{msg}"
+                )
 
         return self
 
     def __exit__(self, *args: Any, **kwargs: Any) -> None:
         if (self.backend is not None) and self._spawn_params is None:
-            self.logger.info(f"Finalized processing group with backend: '{self.backend}'")
+            self._logger.info(  # type: ignore[attr-defined]
+                f"Finalized processing group with backend: '{self.backend}'"
+            )
             idist.finalize()
diff --git a/ignite/utils.py b/ignite/utils.py
--- a/ignite/utils.py
+++ b/ignite/utils.py
@@ -85,12 +85,13 @@ def to_onehot(indices: torch.Tensor, num_classes: int) -> torch.Tensor:
 
 
 def setup_logger(
-    name: Optional[str] = None,
+    name: Optional[str] = "ignite.root.logger",
     level: int = logging.INFO,
     stream: Optional[TextIO] = None,
     format: str = "%(asctime)s %(name)s %(levelname)s: %(message)s",
     filepath: Optional[str] = None,
     distributed_rank: Optional[int] = None,
+    reset: bool = False,
 ) -> logging.Logger:
     """Setups logger: name, level, format etc.
 
@@ -102,6 +103,7 @@ def setup_logger(
         filepath: Optional logging file path. If not None, logs are written to the file.
         distributed_rank: Optional, rank in distributed configuration to avoid logger setup for workers.
             If None, distributed_rank is initialized to the rank of process.
+        reset: if True, reset an existing logger rather than keep format, handlers, and level.
 
     Returns:
         logging.Logger
@@ -127,34 +129,66 @@ def setup_logger(
         # 2020-01-21 12:46:07,358 evaluator INFO: Epoch[1] Complete. Time taken: 00:01:02
         # ...
 
+    Every existing logger can be reset if needed
+
+    .. code-block:: python
+
+        logger = setup_logger(name="my-logger", format="=== %(name)s %(message)s")
+        logger.info("first message")
+        setup_logger(name="my-logger", format="+++ %(name)s %(message)s", reset=True)
+        logger.info("second message")
+
+        # Logs will look like
+        # === my-logger first message
+        # +++ my-logger second message
+
+    Example to change the level of an existing internal logger
+
+    .. code-block:: python
+
+        setup_logger(
+            name="ignite.distributed.launcher.Parallel",
+            level=logging.WARNING
+        )
+
     .. versionchanged:: 0.4.3
         Added ``stream`` parameter.
-    """
-    logger = logging.getLogger(name)
 
-    # don't propagate to ancestors
-    # the problem here is to attach handlers to loggers
-    # should we provide a default configuration less open ?
-    if name is not None:
-        logger.propagate = False
-
-    # Remove previous handlers
-    if logger.hasHandlers():
-        for h in list(logger.handlers):
-            logger.removeHandler(h)
+    .. versionchanged:: 0.5.0
+        Added ``reset`` parameter.
+    """
+    # check if the logger already exists
+    existing = name is None or name in logging.root.manager.loggerDict  # type: ignore
 
-    formatter = logging.Formatter(format)
+    # if existing, get the logger otherwise create a new one
+    logger = logging.getLogger(name)
 
     if distributed_rank is None:
         import ignite.distributed as idist
 
         distributed_rank = idist.get_rank()
 
+    # Remove previous handlers
+    if distributed_rank > 0 or reset:
+
+        if logger.hasHandlers():
+            for h in list(logger.handlers):
+                logger.removeHandler(h)
+
     if distributed_rank > 0:
+
+        # Add null handler to avoid multiple parallel messages
         logger.addHandler(logging.NullHandler())
-    else:
+
+    # Keep the existing configuration if not reset
+    if existing and not reset:
+        return logger
+
+    if distributed_rank == 0:
         logger.setLevel(level)
 
+        formatter = logging.Formatter(format)
+
         ch = logging.StreamHandler(stream=stream)
         ch.setLevel(level)
         ch.setFormatter(formatter)
@@ -166,6 +200,12 @@ def setup_logger(
             fh.setFormatter(formatter)
             logger.addHandler(fh)
 
+    # don't propagate to ancestors
+    # the problem here is to attach handlers to loggers
+    # should we provide a default configuration less open ?
+    if name is not None:
+        logger.propagate = False
+
     return logger
 
 


Test patch:
diff --git a/tests/ignite/test_utils.py b/tests/ignite/test_utils.py
--- a/tests/ignite/test_utils.py
+++ b/tests/ignite/test_utils.py
@@ -120,8 +120,8 @@ def test_setup_logger(capsys, dirname):
 
     def _test(stream):
 
-        trainer.logger = setup_logger("trainer", stream=stream, filepath=fp)
-        evaluator.logger = setup_logger("evaluator", stream=stream, filepath=fp)
+        trainer.logger = setup_logger("trainer", stream=stream, filepath=fp, reset=True)
+        evaluator.logger = setup_logger("evaluator", stream=stream, filepath=fp, reset=True)
 
         assert len(trainer.logger.handlers) == 2
         assert len(evaluator.logger.handlers) == 2
@@ -153,6 +153,32 @@ def _(_):
     logging.shutdown()
 
 
+def _setup_a_logger_and_dump(name, message):
+    logger = setup_logger(name)
+    logger.info(message)
+
+
+def test_override_setup_logger(capsys):
+
+    _setup_a_logger_and_dump(__name__, "test_override_setup_logger")
+
+    source = capsys.readouterr().err.split("\n")
+
+    assert "tests.ignite.test_utils INFO: test_override_setup_logger" in source[0]
+
+    # change the logger level of _setup_a_logger_and_dump
+    setup_logger(name=__name__, level=logging.WARNING, reset=True)
+
+    _setup_a_logger_and_dump(__name__, "test_override_setup_logger")
+
+    source = capsys.readouterr().err.split("\n")
+
+    assert source[0] == ""
+
+    # Needed by windows to release FileHandler in the loggers
+    logging.shutdown()
+
+
 def test_deprecated():
 
     # Test on function without docs, @deprecated without reasons