Building and Exporting Python Logs in Jupyter Notebooks | by Brian Mattis | Towards Data Science

Logging is an important capability within Python to monitor the health of our code. In a way, log files share a lot of the same concepts as the debugger. We can use logs to place breadcrumbs in our code to help determine what’s going on inside. This is especially useful when our script crashes for an unknown reason — if we leave enough breadcrumbs behind we can track the path the code took, and we can focus our attention there. Typically the python debugger is useful when we’re first testing out our code, but at some point we consider our code “working” before we’ve tested every possible input case. If we use logging in the background, we’ll have a great idea of where we went wrong if something breaks down the road. In particular, if we use logging correctly, we can see the exact unique situation that broke our “working” code.
Logs become even more critical when others use our code. When a script fails on another user’s machine, it can be challenging to replicate the error — but the logs can guide us down the path the script followed just before it failed on the user’s system. This is exceptionally useful in creating code that manages all of the “edge” cases (why do you think software always asks for permission to send crash data when you install it?!)
Finally, there’s another awesome use for logs — as trackers of the code development cycle. Imagine that within the first week of release to other users your python script generates 200 errors, 500 warnings, and various other logging flags. As the code is refined and becomes more robust through iteration, we can see the rates of the different issues decrease over time. This gives us an idea of where we are in our development cycle and where to focus our future efforts by tackling the most common errors first. So let’s get started:
Logging in Python when running scripts from the command line is very well documented, but we need to start with the basics before diving into the tricks and tools we can use. First, when we send messages to the logger, we choose which category the information fits into. These are organized by increasing severity, from NotSet to Critical as shown below. What’s powerful about this hierarchy is that when we output the log data, we can select which level to report. In this case, the logger will capture everything at that level and more severe. For example, if we were to chose a level of Warning, then the logger would capture all Warning, Error, and Critical messages. Complete documentation on the standards of when to use each level are found here.
To build this in Python with the logging messages being posted to the console, it looks like this:
import logging#create a loggerlogger = logging.getLogger()# to control the reporting levellogger.setLevel(logging.WARNING)#send two messages to the loggerlogging.info("This is an INFO message")logging.warning("This is a WARNING message")
As the level is set to logging.WARNING we won’t see the info-level message. Setting the level is useful as during first coding we’ll send breadcrumbs in DEBUG and INFO to make sure we’re doing things right. Then, when we think we no longer need the breadcrumbs, we can eliminate them from the log and focus on functionality problems in warnings, errors, and criticals. This can work hand-in-hand with the debugger.
To output to an actual log file instead of the command line, we need to set up the logger using basicConfig() where we can also set the level, like this:
import logginglogging.basicConfig(filename='my_first_log.log', level=logging.INFO)
Unfortunately, this method won’t work when operating in Jupyter Notebooks. At all. The problem lies in that at start-up, a Jupyter Notebook launches an IPython session that has already initiated logging. As a result, logging.basicConfig() won’t work.
Logging is also particularly useful in Jupyter Notebooks, as the debugger can have some serious functionality issues. These debugger issues are avoided by using the enhanced debugger in JupyterLab— but if you’re constrained to Jupyter Notebooks, then logging gives you the flexibility to do what you need to handle a broader range of debug tasks.
To export log messages to an external log file in Jupyter, we’ll need to set up a FileHandler and attach it to the logger:
import logginglogger = logging.getLogger()fhandler = logging.FileHandler(filename='test_log.log', mode='a')logger.addHandler(fhandler)logging.warning('This is a warning message')
It’s important to note the mode selection in the FileHandler setup:
‘a’ — append. All new messages are tacked onto the end of the log file
‘w’ — write. The log file is cleaned out and started fresh each time you run the python script
If all went well, we should have a new ‘test_log.log’ file in the same directory as our Jupyter Notebook.
We can also improve and customize our log outputs to contain more than the text we dictate in the logging messages. By using logging.Formatter() we can add in a host of fields to help us understand more about the messages, such as when the message was generated and by at which line in the code. Here are some of my favorites, with the complete list here:
%(asctime)s — the date/time of the entry
%(levelname)s — the logging level (so you don’t have to spell that out in your error)
%(name)s — name of the logger used (as in, which file generated this message)
%(lineno)s — the line number that generated the message
%(message)s — of course! The message itself!
To attach our formatting to the logger, we add it to the file handler like this:
formatter = logging.Formatter('%(asctime)s - %(name)s - %(lineno)s - %(levelname)s - %(message)s')fhandler.setFormatter(formatter)
We can also add another handler to show this formatting in the console as too. Here, we access the console through a StreamHandler() using sys.stdout and attach it to the logger:
import sysconsoleHandler = logging.StreamHandler(sys.stdout)consoleHandler.setFormatter(formatter)logger.addHandler(consoleHandler)
It gets even better! We can also independently adjust the levels outputted by each handler. For example, we can set our console to report down to the debug level like this: consoleHandler.setLevel(logging.DEBUG)
Important note — The main logger level must be equal to or lower than the level given to the handlers, otherwise it won’t work, as the high-level logger level sets the floor.
We can also pass more than static messages to our logger. Often, if our code is returning warnings or errors, we’ll want to know what user values triggered the event. Luckily, we can simply parse variable values into our messages like common strings using .format().
var1 = 1var2 = 'cat'logging.debug('var1: {}; var2: {}'.format(var1, var2))
With what we’ve learned so far, we can initiate a new logger in Jupyter Notebooks, adjust the reporting level, adjust the formatting, output to file, output to the console, and adjust the level of each output. Here’s a simple snapshot of what it looks like in action, with the console at DEBUG and the export file at INFO:
And here’s what we see in that file_out.log file:
And while it’s nice to have the ability to export the log to a file, sometimes we want to be a bit lazy and not switch over to a text editor to read our output file. To see the log file in Jupyter Notebook, we can just read and print the contents like this:
with open("file_out.log") as log:    print(log.read())
Because the logger is automatically set up as part of IPython when we start a notebook, certain configurations conflict with some of the functionality that exists for python files run in the console. One in particular is the use of the getLogger(__name__) functionality. When our python scripts become more complex, we will often break some functions into separate script files that are later imported as separate modules. We can pass all log messages to a common log file, with the %(name)s formatter telling us which python script generated the message. There’s even a tutorial on how to do this in python running from the console. Unfortunately, this capability is not immediately compatible in Jupyter Notebooks. I haven’t found a solution yet... but for now know that using __name__ will prevent anything from being printed to the log file or the formatted console handler.
As you get deeper into python logs, you’ll find that we can even export log files though email. This is very advantageous — after our code is deployed we can monitor its health from afar. Even better, it only takes a couple of lines of code to add the SMTPHandler. Example code can be found here.
Also, with the ability to set the level for each handler, we can create multiple log outputs simultaneously that report and different levels. For example, we may want one log file that captures every event down to the debug level, but we may also want one that is filtered to only report warning and above — this gives us a simple file to monitor for major problems, and a separate log file of everything for when we want to dive in deeper.
Logs offer a great way to maintain the quality and functionality of our code, especially when we release our python scripts for others to use. It can help flag when our code to acts unexpectedly and leaves a trail of breadcrumbs to show how and why things broke. It’s extraordinarily challenging during development to anticipate every use case and every user input that our scripts are going to receive — logging gives us a way to continually iterate and improve the robustness of our code.
There are a couple of differences when implementing logs in Jupyter Notebooks, but once you understand the modifications you can successfully generate truly useful logs. Thanks for reading, and as always, all of the code shown above can be accessed on my Github here.