The Reusable Python Logging Template For All Your Data Science Apps | by Yash Prakash | Towards Data Science

The perfect way to debug and follow the execution of an app is through well defined, informative and conveniently structured logs.
They are the essential component of any small, medium or large project in any programming language, not just Python.
I started using the Python logging library a couple years ago, and since then, I’ve been consulting countless tutorials and articles online on how to use it effectively and with best possible setup for my projects.
All of them are good at explaining how to set a logging system for a single Python script. However, it is nearly impossible to find one that explains how to set up the Python logging library to be used application-wide, and how to properly integrate and share the logging info comfortably in all your project modules.
In this article, I will be sharing my personal logging template that you can easily use for any project with multiple modules.
I am assuming you know the basics of logging already. Like I said, there are a number of good articles out there to learn from.
Okay, let’s get down to it! 👇
Explaining a new concept should always be done at first in simpler terms with little focus diverted towards background information. With that in mind, let’s go ahead and initialise a simple project for now, shall we?
Create a folder called ‘MyAwesomeProject’. Inside it, create a new Python file called app.py. This will be the starting point of our app. I will be using this project to construct a simple working example of the template I’m talking about.
Go ahead and open your project in VSCode (or any editor you prefer).
Now, let’s create a new module for the app level logging setup. We’ll call it logger.
And we’re done with this part.
This is the main part of our template. We create a new file called logger.py.
Let’s define a root logger and use it for initialising our app level logger. Time for some code!
A few imports and our app name:
import loggingimport sysAPP_LOGGER_NAME = 'MyAwesomeApp'
The function we will be calling in our app.py:
def setup_applevel_logger(logger_name = APP_LOGGER_NAME, file_name=None):     logger = logging.getLogger(logger_name)    logger.setLevel(logging.DEBUG)    formatter = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")    sh = logging.StreamHandler(sys.stdout)    sh.setFormatter(formatter)    logger.handlers.clear()    logger.addHandler(sh)    if file_name:        fh = logging.FileHandler(file_name)        fh.setFormatter(formatter)        logger.addHandler(fh)    return logger
We will be defining our logger with the predefined DEBUG level and use a Formatter to structure our logging messages. We then assign it to our stream handler in order to write messages to the console.
Next, we also make sure to include a file in which we can additonally store all our log messages too. This is done via the logging FileHandler.
Finally, we return the logger.
Another function is needed which will make sure our modules can call the logger when needed. Define a get_logger function.
def get_logger(module_name):       return logging.getLogger(APP_LOGGER_NAME).getChild(module_name)
Also, in order to use this module as a package, we can optionally make a folder called logger and put this file inside it. If we do that, we’ll also need to include a ___init__.py file in the folder and do
from .logger import *
to make sure we can import our module from the package.
Great! The main setup is done!
A simple module to test our logger can be done to understand the template better. Let’s define it a simple module.py.
import loggerlog = logger.get_logger(__name__)def multiply(num1, num2): # just multiply two numbers    log.debug("Executing multiply function.")    return num1 * num2
Awesome! This module now has access to the logger and should display the message with the appropriate module name.
Let’s test it now!
Now, we construct the app.py.
import loggerlog = logger.setup_applevel_logger(file_name = 'app_debug.log')import mymodulelog.debug('Calling module function.')mymodule.multiply(5, 2)log.debug('Finished.')
Notice how we import our module after initialising the logger? Yes, it’s necessary.
Now, verify that your directory contains these files:
Finally, run the script simply via:
python3 app.py
You should get an output like:
And your directory structure should also change to include the new log file. Go ahead and inspect its contents!
This is how you can easily integrate a working logging setup in your projects. 😄.
It’s simple and you can easily extend it to include numerous hierarchies between different modules, capture and format exceptions with the logger, configure it in a more advanced manner using a dictConfig, and so on. The possibilities are endless!
The code repository is located here. It also contains the resources for all my other articles too. You can go ahead and ⭐️ and bookmark it!
If you liked this article, every week I put out a story in which I share little pieces of knowledge from the world of data science and programming in general. Follow me to never miss them!
Another couple of articles of mine that you might find useful: