8 Advanced Python Logging Features that You Shouldn’t Miss | by Xiaoxu Gao | Towards Data Science

Logging is a very important unit in software development. It helps developers have a better understanding of the execution of the program and reason about defects and unexpected failures. A log message can store information like the current status of a program or where the program is running. If an error occurs, developers can quickly find the line of code that causes the issue and act upon that.
Python provides a quite powerful and flexible built-in logging module with many advanced features. In this article, I want to share 8 advanced features that will benefit us in developing the software.
Before looking at more advanced features, let’s make sure we have a basic understanding on logging module.
Logger
The instance that we create to generate logs is called logger. It’s instantiated via logger = logging.getLogger(__name__). The best practice is to use __name__ as the logger name which includes the package name and module name. The name will appear in the log message which helps the developers quickly find out where the log is generated.
Formatter & Handler
Every logger has a number of configurations that can be modified. More advanced configurations will be discussed later, but the most common ones are formatter and handler.
Formatter specifies the structure of the log message. Each log message is a LogRecord object with a number of attributes (module name is one of them). When we define a formatter, we can decide how the log message should look like with these attributes and possibly with customized attributes. The default formatter looks like this:
severity:logger name:message# e.g: WARNING:root:Program starts!
A customized formatter with more attributes can look like this:
"%(asctime)s - [%(levelname)s] -  %(name)s - (%(filename)s).%(funcName)s(%(lineno)d) - %(message)s"# 2020-07-26 23:37:15,374 - [INFO] -  __main__ - (main.py).main(18) - Program starts!
Handler specifies the destination of log messages. A log message can be sent to more than one destination. The logging module actually provides quite a few standard handlers. The most popular ones are FileHandler which sends logs to a file and StreamHandler which sends logs to streams such as sys.stderr or sys.stdout. A logger instance supports 0 or more handlers. If no handlers are defined, then it will send logs to sys.stderr. If more than 1 handler are defined, then the destination is dependent on the level of the log message and and the level of the handler.
For example, I have a FileHandler with level WARNING and a StreamHandler with level INFO. If I write an error log message, then the message will be sent to both sys.stdout and the log file.
Example:
In this example, we created a main.py, package1.py and app_logger.py. app_logger.py contains a function get_logger that returns a logger instance. The logger instance comes with a custom formatter and 2 handlers: StreamHandler with INFO level and FileHandler with WARNING level. It is important to set the basic level to INFO or DEBUG (the default logging level is WARNING), otherwise any logs lower than WARNING level will be filtered out. Both main.py and package1.py creates uses get_logger to create their own loggers.
WARNING logs are sent to both console output (sys.stdout) and the log file, but INFO logs are only sent to console output. If you can fully understand what is going on in this example and why then we are good to go for more advanced features.
As I mentioned previously, the LogRecord has a number of attributes, developers can pick up the most important attributes and put them in the formatter. In addition to that, the logging module also provides the possibilities to add user-defined attributes to the LogRecord.
One way is to use LoggerAdapter. When you create an adapter, you pass the logger instance and your attributes (in dictionary) to it. This class provides the same interface as Logger, so you can still call methods like logger.info.
New attribute with fixed value
If you want to have a kind of fixed value attribute in the log message such as application name, then you can use the default LoggerAdapter class and get attribute value when you create the logger. Don’t forget to add this attribute in the formatter, you are free to choose the position of it. In the following code, I add the attribute app, this value is defined when I create the logger.
New attribute with dynamic value
In other situations, you might want dynamic attributes, for instance, a kind of dynamic ID. Then you can extend the basic LoggerAdapter and create your own. process() method is where the extra attributes are appended to the log message. In the following code, I add a dynamic attribute id which can be different in each log message. In this case, you don’t need to add the attribute in the formatter.
Another way to add dynamic user-defined attributes is using custom Filter. Filters provide extra logic to determine which log messages to output. It is a step after checking the basic logging level, but before passing the log message to handlers. In addition to determining whether the log message should move forward, we can also insert new attributes in the method filter().
In this example, we add a new attribute color in filter() whose value is determined based on the level name of the log message. In this case, the attribute name again should be added in the formatter.
Logging module is actually implemented in a thread-safe way, so we don’t need extra efforts. The code below shows that MainThread and WorkThread are sharing the same logger instance without race condition problem. There is also a built-in attribute threadName for the formatter.
Under the hood, the logging module uses threading.RLock() pretty much everywhere. The differences between RLock from Lock are:
Lock can only be acquired once and cannot be acquired anymore until it’s released. On the other hand, RLock can be acquired multiple times before release, but it should be released the same number of times.Lock can be released by any thread, but RLock can only be released by the same thread which acquires it.
Lock can only be acquired once and cannot be acquired anymore until it’s released. On the other hand, RLock can be acquired multiple times before release, but it should be released the same number of times.
Lock can be released by any thread, but RLock can only be released by the same thread which acquires it.
Any handler which extends from class Handler has a method handle() to emit records. This is a code block of Handler.handle(). As you can see, the handler will acquire and release the lock before and after emitting the record. emit() method can be implemented differently in different handlers.
Although logging module is thread-safe, it’s not process-safe. If you want multiple processes to write to the same log file, then you have to manually take care of the access to your file. According to the logging Cookbook, there are several options.
QueueHandler + “consumer” process
One option is using QueueHandler. The idea is to create a multiprocessing.Queue instance and share it among multiple processes. In the example below, we have 2 “producer” processes which send logs to the queue and a “consumer” process reading logs from the queue and write them into the log file.
The logs in the queue will probably have different levels, therefore in log_processor, we use logger.log(record.levelno, record.msg) instead of logger.info() or logger.warning(). In the end, we send a signal to let log_processor stop. Sharing a queue instance among multiple processes or threads is not a new thing, but logging module kinds of helping us handle this situation.
QueueHandler + QueueListener
In logging.handlers module, there is a special class called QueueListener. This class creates a listener instance with a log message queue and a list of handlers to handle log records. QueueListener can replace the listener process we created in the previous example with less code.
SocketHandler
Another solution provided by the Cookbook is sending logs from multiple processes to a SocketHandler and have a separate process which implements a socket server that reads logs and send to the destination. The documents has a quite detailed implementation.
All of these solutions basically follow the same principle: sending logs from different processes to a centralized place, either a queue or a remote server. The receiver on the other side is responsible for writing log records into destinations.
So far, we’ve mentioned a number of handlers implemented by logging module. Another useful built-in handler is NullHandler. The implementation of NullHandler is basically nothing. Nevertheless, it helps developers distinguish library logs from application logs.
This is the implementation of NullHandler.
Why do we need to distinguish library logs from application logs?
According to the author of logging module, Vinay Sajip:
A third party library which uses logging should not spew logging output by default which may not be wanted by a developer/user of an application which uses it.
The best practice is to not emit library logs by default and let the user of the library determine whether they want to receive and handle logs in the application.
As a library developer, we only need one line of code inside __init__.py to add NullHandler. In subpackages and submodules, the loggers keep as usual. When we install this package in our application via pip install, we won’t see the logs from the library by default.
The way to make these logs visible is adding handlers to the library logger in your application.
# your application logging.getLogger("package").addHandler(logging.StreamHandler())
If the library doesn’t use NullHandler, but you want to disable the library logs, then you can set logging.getLogger("package").propagate = False. If propagate is set to False, then logs will not be passed to the handlers.
RotatingFileHandler supports log file rotation, which gives the handler the ability to rotate log files based on its maximum size. 2 parameters should be defined here: maxBytes and backupCount. maxBytes tells the handler when to rotate the log. backupCount is the number of log files, each extended log file has a suffix “.1”, “.2” at the end of filename. If the current log message is about to let the log file exceed maximum size, then the handler will close the current file and open the next file.
This is an example very much similar to the Cookbook. You should get 6 log files.
Another file rotation handler is TimeRotatingFileHandler, which allows developers to create rotating logs based on elapsed time. The time conditions include: second, minute, hour, day, w0-w6 (0=Monday) and midnight (rollover at midnight).
In the following example, we rotate log file every second with 5 backup files. Each backup file has a timestamp as the suffix.
In many cases, we use logger.error() or logger.exception() when handling exceptions. But what if the logger itself raises an exception, what would happen to the program? Well, it depends.
The logger error is handled when the handler calls emit(), which means any exception related to formatting or writing is caught by the handler rather than being raised. More specifically, handleError() method will print out the trackback to stderr and the program will continue. If you have a custom handler extending from Handler class, you can implement your own handleError().
In this example, the second log message has too many arguments. So in the console output, we received trackback and the program could still continue.
However, if the exception happens outside of emit(), then the exception might be raised and the program would stop. For example, in the code below, we add an extra attribute id in logger.info() without handling it in LoggerAdapter. This error is not handled and causes the program to stop.
The last point I want to share is about configuring your logger. There are 3 ways to configure a logger.
use code
The most straightforward option is to use code to configure your logger, just like all the examples we’ve seen so far in this article. But the disadvantage of this option is that any modification requires a change in the source code.
use dictConfig
The second option is to write configuration in a dictionary and use logging.config.dictConfig to read it. You can also save the dictionary to a JSON file and read from there. The advantage is that it can be loaded as an external configuration, but it can be error-prone due to its structure.
use fileConfig
Last but not least, the third option is to use logging.config.fileConfig. The configuration is written into a separate .ini file.
It’s possible to update the configuration during the runtime via a configuration server. Cookbook shows an example of both client side and server side. The configuration is updated through a socket connection, and on the client side, we use c = logging.config.listen(PORT) c.start() to receive the latest configuration.
I hope these logging tips and tricks can help you set up a nice logging framework around your application without hurting the performance. If you have anything to share, please leave your comments below!