Better ways of Logging with Python
By: Mike Wolfe
Tags: Python, Sql, Programming, MySQL, Object Oriented

Better ways of Logging with Python

Can there be a better way?

Photo by Jan Antonin Kolar on Unsplash

Back Story

During the summer of 2016, I began my first internship at an awesome company. The part that wasn’t awesome was that file logging was being used for a couple of Monolithic applications (These have since been changed). As you can imagine, it was quite a pain having to search through those files to figure out what occurred at certain times. This got me thinking about how I could have better logging with my applications. Enter Python.

Initially, the Python “logging” module was what I had in mind. There is an excellent tutorial out there that explains the functionality of the module. It has some cool features such as different levels of logging such as Debug, Info, Error, etc. Unfortunately, the module outputs logs to a file, which is not what I was looking for. I was hoping that there would be a way to output to a database.

A Potential Solution?

After taking some time to think about what I was looking for, I created a database and a couple of tables. These tables currently hold the bare minimum for information, but have plenty of room for expansion.

The Application table houses all of the different applications that will use the logging database. The Info, Error, and Debug tables are the different logging levels that events can be logged to. Essentially, these tables follow the same idea as the Python logging module. The last piece I created, was a Python class that can be imported into another file.

To import the file, I simply did this:

from my_logger import MyLogger

To create the class object for our newly imported logging module:

logger = MyLogger("Application Name", "mysql://<USER>:<PASSWORD>@<HOST>/<DATABASE>")

After that, logging can begin!

logger.LogInfo("Logging info here")

logger.LogDebug("Logging debuging here")

logger.LogError("Logging errors here")

Here is what the output looks like in the database:

Application table.

Info table.

Debug table.

Error table.

The class isn’t anything complicated. It requires an application name and connection string to be initialized. Once provided, you can utilize whichever logging level you need. The class will then insert those logs into their corresponding tables.

Enhancements

Thinking about some ways to improve upon this, it would be nice to create an API (Application Programming Interface) that would integrate with the Python class. The perk would be that any application would be able to use this logging database. It may sound hypocritical of me, but outputting database connection errors to a file would be pretty helpful. This way, you will have some vision as to why logs are not getting inserted into the database.

One part that I did not cover in this article was creating the table classes. In our example above, it was the log_models file. These classes are required for SQLAlchemy to work properly. It is a fairly easy task and you find out more about them here. It’s worth mentioning that these table classes require a separate connection string. It would be more convenient to only have to pass this connection string once through the class. We would have to figure out a way to pass it through both the logging class file and the table classes file. It would be even more proper if those connection strings were an environment variable.

Conclusion

On a final note, this logging class and tables is just something simple to get us started. This is by no means the final product. Feel free to make any enhancements you like! Also, I would be more than happy to hear any suggestions you may have for the class, or if you have a different way of logging that doesn’t require outputting to a file. Happy logging and cheers!