Repository: cocotb/cocotb

Issue:
Why is the default logger overridden?
https://github.com/cocotb/cocotb/blob/ba0fb097efbd2939ad6a961ee8d02793104ef8f7/cocotb/__init__.py#L60

How about instead of forcing users to use the cocotb logger through `logging`, we just extend `cocotb.log` to act more like the logging module and get users to use that instead?


Discussion:
This is already a lot simpler than it used to be (see #984 )

If you look at the class, it doesn't really do very much. The most important thing is the _logFromC method:

https://github.com/cocotb/cocotb/blob/17ce4e51da43b6d2d1b5834d629374ed5535a8d8/cocotb/log.py#L54-L95

I suppose that could be a free function, there's not a particularly strong argument for it to be a method.
That's not really what I'm asking. I'm asking why that quoted line is there. Is there any reason to set the logger class in `logging`? I see in a lot of files there are lines like

```python
import logging
testlog = logging.getLogger('cocotb.test')
```

Why override `logging` when you could instead just set up `cocotb.log` to act just like the `logging` module, so that you could do

```python
import cocotb.log
cocotb.log.getLogger('test')
```

Why replace the world, instead of coexist? Sometimes people want to use the `logging` module for things that aren't a simulation status/debugging logfile.
> Is there any reason to set the logger class in logging?

Here's why we do it today:

* To set a default logging formatter / handler
* To provide a hook to allow our C code to call into the python logging stuff (removed in gh-1213)
* To provide `my_logger.logger = my_logger`, for backwards compatibility with the time when `SimLog` returned an opaque wrapper around a regular logger.
* To provide `my_logger.colour = cocotb.utils.want_color_output()`

The first two of these would be possible without setting the logging class. The third isn't, but arguably isn't very useful.
The last point is quite useful for me at least.
@stuarthodgson: How would you feel about deprecating `log.colour` in favor of just calling `cocotb.utils.want_color_output()` directly?

@ktbarrett, back to your original question:

> How about instead of forcing users to use the cocotb logger

Are you complaining that users can't now specify their own handler class?  There's nothing to stop a user doing:
```
@logging.setLoggerClass
class MyLog(logging.getLoggerClass()):
   pass  # this will work both with and without cocotb
```

Or are you complaining about the fact we set a default handler / formatter for all loggers?
> Or are you complaining about the fact we set a default handler / formatter for all loggers?

Yes. To be less round-about, we have our own logger that we use. We want to keep the format consistent with the rest of our software. We used to maintain a fork of cocotb that had the changes in `log.py` we wanted, and they were rather extensive. With the not-so-recent changes to the logger, which were also rather extensive, merging is not possible. We will have to start from scratch, so ignore my complaints.
Likely we will provide our own logger class and `SimLog` implementation and just change the import on this line.
https://github.com/cocotb/cocotb/blob/17ce4e51da43b6d2d1b5834d629374ed5535a8d8/cocotb/__init__.py#L42
I think I'd like to keep this open - the points about the class being unnecessary are still valid, and we could look at making the default config disableable by an env var.

You might find that you don't need to fork cocotb at all, but instead can just revert its setting changes at startup.

If you're able to share it, an outline of your old and/or new patch would be insightful to working out the best way for cocotb to stay out of your way.
I think the best way forward is to move logger setup out of `__init__.py` and into the `cocotb.log` module. `__init__.py` would first try to load a user's custom logger module (by an established name) and if `ModuleNotFoundError`, it would load `cocotb.log` as a fallback. A custom logger will obviously have to play nicely with how cocotb currently uses the logger. Maybe some documentation on required interfaces.

I'd like to see this pattern be applied to the embed_gpi/cocotb module entry point as well, so we could overload the entire (or at least part) of the setup to support launching into IPython.
> would first try to load a user's custom logger module

I think we just leave this to the python logging infrastructure.

> I'd like to see this pattern be applied to the embed_gpi/cocotb module entry point as well, so we could overload the entire (or at least part) of the setup to support launching into IPython.

Filed #1225 for that to keep discussion here focused

> Maybe some documentation on required interfaces.

Once #1213 is in, the answer will basically be "there is no cocotb logging interface, just configure the python logging module as you see fit".

We need the ability to inject a Logger class before the cocotb, gpi, scheduler, and regression loggers are constructed, or there will be inconsistent logging behavior, no? Currently no user code is run until the user's test module is loaded. 

Resolving patch:
diff --git a/cocotb/__init__.py b/cocotb/__init__.py
--- a/cocotb/__init__.py
+++ b/cocotb/__init__.py
@@ -39,8 +39,8 @@
 import warnings
 
 import cocotb.handle
+import cocotb.log
 from cocotb.scheduler import Scheduler
-from cocotb.log import SimBaseLog, SimLog
 from cocotb.regression import RegressionManager
 
 
@@ -56,36 +56,42 @@
 
 # GPI logging instance
 if "COCOTB_SIM" in os.environ:
-    import simulator
-    logging.basicConfig()
-    logging.setLoggerClass(SimBaseLog)
-    log = SimLog('cocotb')
-    level = os.getenv("COCOTB_LOG_LEVEL", "INFO")
-    try:
-        _default_log = getattr(logging, level)
-    except AttributeError as e:
-        log.error("Unable to set loging level to %s" % level)
-        _default_log = logging.INFO
-    log.setLevel(_default_log)
-    loggpi = SimLog('cocotb.gpi')
-    # Notify GPI of log level
-    simulator.log_level(_default_log)
+
+    def _reopen_stream_with_buffering(stream_name):
+        try:
+            if not getattr(sys, stream_name).isatty():
+                setattr(sys, stream_name, os.fdopen(getattr(sys, stream_name).fileno(), 'w', 1))
+                return True
+            return False
+        except Exception as e:
+            return e
 
     # If stdout/stderr are not TTYs, Python may not have opened them with line
     # buffering. In that case, try to reopen them with line buffering
     # explicitly enabled. This ensures that prints such as stack traces always
     # appear. Continue silently if this fails.
-    try:
-        if not sys.stdout.isatty():
-            sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 1)
-            log.debug("Reopened stdout with line buffering")
-        if not sys.stderr.isatty():
-            sys.stderr = os.fdopen(sys.stderr.fileno(), 'w', 1)
-            log.debug("Reopened stderr with line buffering")
-    except Exception as e:
-        log.warning("Failed to ensure that stdout/stderr are line buffered: %s", e)
+    _stdout_buffer_result = _reopen_stream_with_buffering('stdout')
+    _stderr_buffer_result = _reopen_stream_with_buffering('stderr')
+
+    # Don't set the logging up until we've attempted to fix the standard IO,
+    # otherwise it will end up connected to the unfixed IO.
+    cocotb.log.default_config()
+    log = logging.getLogger(__name__)
+
+    # we can't log these things until the logging is set up!
+    if _stderr_buffer_result is True:
+        log.debug("Reopened stderr with line buffering")
+    if _stdout_buffer_result is True:
+        log.debug("Reopened stdout with line buffering")
+    if isinstance(_stdout_buffer_result, Exception) or isinstance(_stderr_buffer_result, Exception):
+        if isinstance(_stdout_buffer_result, Exception):
+            log.warning("Failed to ensure that stdout is line buffered", exc_info=_stdout_buffer_result)
+        if isinstance(_stderr_buffer_result, Exception):
+            log.warning("Failed to ensure that stderr is line buffered", exc_info=_stderr_buffer_result)
         log.warning("Some stack traces may not appear because of this.")
 
+    del _stderr_buffer_result, _stdout_buffer_result
+
     # From https://www.python.org/dev/peps/pep-0565/#recommended-filter-settings-for-test-runners
     # If the user doesn't want to see these, they can always change the global
     # warning settings in their test module.
diff --git a/cocotb/decorators.py b/cocotb/decorators.py
--- a/cocotb/decorators.py
+++ b/cocotb/decorators.py
@@ -229,7 +229,13 @@ def __init__(self, fn):
             logging.Handler.__init__(self, level=logging.DEBUG)
 
         def handle(self, record):
-            self.fn(self.format(record))
+            # For historical reasons, only logs sent directly to the `cocotb`
+            # logger are recorded - logs to `cocotb.scheduler` for instance
+            # are not recorded. Changing this behavior may have significant
+            # memory usage implications, so should not be done without some
+            # thought.
+            if record.name == 'cocotb':
+                self.fn(self.format(record))
 
     def __init__(self, inst, parent):
         self.error_messages = []
diff --git a/cocotb/log.py b/cocotb/log.py
--- a/cocotb/log.py
+++ b/cocotb/log.py
@@ -51,20 +51,51 @@
 _FUNCNAME_CHARS = 31  # noqa
 
 
-class SimBaseLog(logging.getLoggerClass()):
-    def __init__(self, name):
-        super(SimBaseLog, self).__init__(name)
+def default_config():
+    """ Apply the default cocotb log formatting to the root logger.
 
-        # customizations of the defaults
-        hdlr = logging.StreamHandler(sys.stdout)
+    This hooks up the logger to write to stdout, using either
+    :class:`SimColourLogFormatter` or :class:`SimLogFormatter` depending
+    on whether colored output is requested.
+
+    The logging level for cocotb logs is set based on the
+    :envvar:`COCOTB_LOG_LEVEL` environment variable, which defaults to ``INFO``.
+
+    If desired, this logging configuration can be overwritten by calling
+    ``logging.basicConfig(..., force=True)`` (in Python 3.8 onwards), or by
+    manually resetting the root logger instance, for which examples can be
+    found online.
+    """
+    # construct an appropriate handler
+    hdlr = logging.StreamHandler(sys.stdout)
+    if want_color_output():
+        hdlr.setFormatter(SimColourLogFormatter())
+    else:
+        hdlr.setFormatter(SimLogFormatter())
+
+    logging.setLoggerClass(SimBaseLog)  # For backwards compatibility
+    logging.basicConfig()
+    logging.getLogger().handlers = [hdlr]  # overwrite default handlers
+
+    # apply level settings for cocotb
+    log = logging.getLogger('cocotb')
+    level = os.getenv("COCOTB_LOG_LEVEL", "INFO")
+    try:
+        _default_log = getattr(logging, level)
+    except AttributeError:
+        log.error("Unable to set logging level to %r" % level)
+        _default_log = logging.INFO
+    log.setLevel(_default_log)
+
+    # Notify GPI of log level, which it uses as an optimization to avoid
+    # calling into Python.
+    if "COCOTB_SIM" in os.environ:
+        import simulator
+        simulator.log_level(_default_log)
 
-        if want_color_output():
-            hdlr.setFormatter(SimColourLogFormatter())
-        else:
-            hdlr.setFormatter(SimLogFormatter())
 
-        self.propagate = False
-        self.addHandler(hdlr)
+class SimBaseLog(logging.getLoggerClass()):
+    """ This class only exists for backwards compatibility """
 
     @property
     def logger(self):


Test patch: