How to Setup Logging for your Python Notebooks in under 2 Minutes | by Dr. Gregor Scheithauer | Towards Data Science

Gregor is a data scientist who loves to solve data riddles. One day, he is given a new project and wants to jump into this new task immediately. He is prepared and a master of his favorite coding tool of choice, knows which packages to use, and already he forms an idea of how to structure the data project. Logging is not on his mind. Not at the start, that is. It dawns on him when most of the code is written and there is only one bug that he tries to pinpoint and find in his code. Once he smashed his bug and rewritten his flawed code, he realizes his code is cluttered with print() statements. He needs to get them out before passing on his solution.
Maybe you can relate to this little scenario. But I learned that there is an easy fix to my shortcoming. I took a bit of time and looked into the standard python logging framework and I will explain it happily here for my future self and, of course, for you.
I will provide us with a copy-paste-logging-code-template to make logging simple. Also, I will share additional information about the three most important logging concepts. The article ends with relevant articles about logging that I found during my research.
Here is the cop- paste template for logging to your console. Find a logging template to set up a log file in section 3 (log handler).
Additional tips
You may change log levels based on your task or state of development, no need to delete print statements after a debugging sessionIt is possible to log statements into different outputs. A file for audit-purposed logging, a console output during development with a debug level, or a console output during production with only warnings and error messages.Try to write meaningful logging statements for other developers and admins that run your code.Whenever possible provide context information, such as a (hashed) user id, session id, so that debugging or error handling is possible
You may change log levels based on your task or state of development, no need to delete print statements after a debugging session
It is possible to log statements into different outputs. A file for audit-purposed logging, a console output during development with a debug level, or a console output during production with only warnings and error messages.
Try to write meaningful logging statements for other developers and admins that run your code.
Whenever possible provide context information, such as a (hashed) user id, session id, so that debugging or error handling is possible
If you already work with Python or you are familiar with programming this code block might be already sufficient to work with. To understand and remember it more easily, I try to think of it with WHAT — WHERE — HOW.
The WHAT refers to the logger itself together with the different log levels.
The WHERE refers to where the logging output or rather handler should go. This might include your console and a file.
The HOW refers to how the log statements should look like or rather what should be included? This is done by the Formatter.
WHAT — Log levelsBelow you find an overview table with the five log levels. If you follow the GitHub gist you will also see that for each level I added in which application life cycle you could use the log levels.
If you set your log level to debug(logger.setLevel(logging.DEBUG)) then all logging statements that you wrote (DEBUG, INFO, WARNING, ERROR, CRITICAL) will be logged. If you set your log level to WARNING (logger.setLevel(logging.WARNING)) then only the logging statements (WARNING, ERROR, CRITICAL) will be logged.
This will give you the advantage of letting all your logging statements in the code, even when you go into production. You will only need to change the log level.
During development, you might use one of the two log levels DEBUG or INFO. During debugging you will use DEBUG. When you deploy your code into production you should set the log level to WARNING. Below you will find two code examples with different log levels, and hence, different logging outputs.
EXAMPLE 01 — Log level set to debug
DEBUG - User User XYZ provided the numbers 100 and 0 for calculationERROR - Calculation was unsuccessful for user User XYZ with the inputs 100 and 0ERROR - division by zero
EXAMPLE 02 — Log level set to warning
ERROR - Calculation was unsuccessful for user User XYZ with the inputs 100 and 0ERROR - division by zero
Please note, that in the second example the debug log statement is omitted since the log level is set to warning, hence only warnings, errors, and critical statements are displayed.
WHERE — Log handlersIn my own experience, I use the file and the console (stream) handlers. However, the python logging documentation lists 15 different logging handlers. Please check it out for your specific needs. How to set up the StreamHandler to log in to the console is shown above.
I will show you in this example how to set up a FileHandler to create a logfile next to the console output.
HOW — format your log statementMost of the examples I found where formatted in this way, but of course, you may adapt the configuration as you see fit:
Date-time informationName of the logger (the name you provided with the getLogger-function)The log level of the log statementThe log message itself.
Date-time information
Name of the logger (the name you provided with the getLogger-function)
The log level of the log statement
The log message itself.
Logging package documentation
An example from the package documentation
Another introduction into logging
13 tips for logging
Stop Using Print to Debug in Python. Use Icecream Instead by Khuyen Tran
In this article, I try to trick you into using the logging package in your next Python project. I offer a copy-paste-logging-code-setup. Then I explain three important logging concepts: log levels, log handlers, and log formatter. Finally, I provide a list with further information.
Please feel free to contact me with any questions and comments. Thank you. Find more articles from me here:
Learn how I plan my articles for MediumLearn how to write clean code in Python using chaining (or pipes)Learn how to analyze your LinkedIn data using RLearn how to create charts in a descriptive way in Python using grammar of graphics
Learn how I plan my articles for Medium
Learn how to write clean code in Python using chaining (or pipes)
Learn how to analyze your LinkedIn data using R
Learn how to create charts in a descriptive way in Python using grammar of graphics
Gregor Scheithauer is a consultant, data scientist, and researcher. He is specialized in the topics of Process Mining, Business Process Management, and Analytics. You can connect with him on LinkedIn, Twitter, or here on Medium. Thank you!