How to Implement Logging in Your Python Application
By: Leo Brack
Tags: Programming, Software Development, Python, Startup, Data Science

Enter Python’s Logging Module

Fortunately, the importance of logging is not a new phenomenon. Python ships with a ready-made logging solution as part of the Python standard library. It solves all the aforementioned problems with using print . For example:

Automatically add context, such as line number and timestamps to logs.

It’s possible to update our logger at runtime by passing a configuration file to the app.

It is easy to customise the log severity and configure different logging levels for different environments

Let’s try it out and set up a very basic logger:

Running this gives:

INFO:__main__:Getting some docs...

INFO:__main__:Doc count 2

INFO:__main__:Finished

Easy peasy!

Here, we have imported the logging module from the Python standard library. We then updated the default basic log level to log INFO messages. Next, logger = logging.getLogger(__name__) instantiates our logging instance. Finally, we passed an event to the logger with a log level of INFO by calling logger.info("") .

At first glance, this output might appear suspiciously similar to using print() . Next, we’ll expand our example logger to demonstrate some of the more powerful features that the Python standard logging module provides.

Log levels

We can configure the severity of the logs being output and filter out unimportant ones. The module defines five constants throughout the spectrum, making it easy to differentiate between messages. The numeric values of logging levels are given in the following table:

Logging levels from Python’s documentation.

It’s important not to flood your logs with lots of messages. To achieve concise logs, we should be careful to define the correct log level for each event:

logger.critical("Really bad event" )

logger.error("An error")

logger.warning("An unexpected event")

logger.info("Used for tracking normal application flow")

logger.debug("Log data or variables for developing")

I tend to use the debug level to log the data being passed around the app. Here is an example of using three different log levels in the few lines of code responsible for sending events to Kafka:

Formatting logs

The default formatter of the Python logging module doesn’t provide a great amount of detail. Fortunately, it is easy to configure the log format to add all the context we need to produce super-useful log messages.

For example, here we add a timestamp and the log level to the log message:

formatter = logging.Formatter('%(asctime)s - %(levelname)s - %(message)s')

It’s best practice to add as much context as possible to your logs. This can easily be achieved by adding structured data to the log message’s metadata. For example, you may have scaled your application to run with multiple workers. In this case, it might be important to know which worker was logging each event when you’re debugging, so let’s add a worker ID context to the log metadata:

# Create the log formatter formatter = logging.Formatter( '%(asctime)s - %(worker)s %(levelname)s - %(message)s')

handler.setFormatter(formatter) logger.info('Querying database for docs...', extra={'worker':

'id_1'})

The output becomes:

2020-09-02 22:06:18,170 - id_1 - INFO - Querying database for docs...

Log handlers

Now that we have perfectly formatted logs being fired at us from all over our application code, we need to consider where those logs are ending up. By default, the logs are being written to stdout , but Python’s logging module provides us with the functionality to push logs to alternative locations. For example, to save logs to the example.log file on disk:

# create a file handler

handler = logging.FileHandler('example.log')

handler.setLevel(logging.INFO)

There are several types of handlers that can be used. For the complete list, see the documentation for handlers. It is also possible to define custom logging handlers for different use cases. For example, this library defines a log handler for pushing logs to Slack!

To summarise. We’ve set up the Python standard logging module and configured it to log to different locations with custom log formats. You can find the final code for the example logger below: