15 Surprisingly Useful Python Base Modules | by Emmett Boudreau | Towards Data Science

Earlier this month, I went in-depth on some seriously powerful and surprisingly useful functions that are in the base of the Python programming language. These standard library modules and Python functions are pretty awesome, but a fortunate thing is that they are all only only a blink in the eye when it comes to what the Python programming language has to offer. If you haven’t found the time to read my previous article where I outline some other standard library tools that I think might even come in handy more often than these, you can check it out here:
towardsdatascience.com
Despite the number of fabulous features I outlined in the previous iteration, there is a lot more that the Python standard library has to offer for those programmers that might be determined enough to become a wiz with it. The real strength of Python lies in the simplicity of its base and the abilities it has right out of the box, particularly to manipulate data and work with other languages and scripts. This is of course reflected very much in the packages it provides, which often provide the functionality for working with external code, text, and data. Data manipulation is only a small glint of what the Python programming language’s standard library is actually capable of. That being said, there are several examples of packages that can be used to take the standard Python experience to the next level just with a little bit of knowledge. Here are 15 more of my favorites that I think most Python programmers might want to take a look into!
notebook
One type that certainly proves Python’s somewhat scientific and numerical intentions is the complex type. This type has been in Python for a while, and allows both imaginary and real numbers to be stored as one complete value. This is most often made apparent by the addition of numbers in the imaginary dimension to a float, for example:
5.5 + 8.2im
Where:
5.5 is a real float.
8.2 is an imaginary float.
However, beyond just holding numbers, complex floats are essentially useless without the proper operations to support them. This is where the cmath module comes in to save the day. Not only does the cmath have support for basic operations like square roots and logarithmic functions, but it also provides more advanced functions like phase, polar, or x.
This module is almost certainly a must for any Python user that might interact with complex numbers. Not only will it provide a sufficient enough back-bone to do just about anything with numbers that are at least partially imaginary, it can also provide a lot more information on polar coordinates in regards to those numbers. For my examples, I chose to utilize the exp(), log() and phase() methods from this module. However, it goes without saying that there are a lot more options available inside of this fantastic tool that are most certainly worth checking out!
In order to use cmath, we will first need to import it. I personally usually find myself aliasing it as cm, but I must confess I am not certain if this is a convention or not:
import cmath as cm
Next, we will create a complex number we can evaluate with:
z = 5.5 + 8.2j
Finally, we will take our new complex value and use some of the fabulous methods from the cmath package on our new variable!
cm.phase(z)cm.log(z)cm.exp(z)
The next incredibly valuable tool from the Python standard library is a little module called itertools. I cannot recommend the use of this module enough, because it really does seem to come in handy a lot more frequently than I ever would have thought it would. The itertools module can be used to quite effectively and easily create better and sometimes unique iteration that would simply take a whole lot more time without the use of the module.
This module is basically a composition of some of the industries leading tools for iteration comprised into a purely Pythonic package. In a lot of scenarios, these tools are simply covered in Python by combining the use of other Pythonic features. A great example of this is mentioned in the documentation, as well. The SML programming language provides a tabulation tool, the tabulate() method, which can be recreated in the Python programming language using the map and count functions already available in the standard library.
Needless to say, this module is certainly one that is worth having! The real value of this module comes when working with iteration, because it can make a lot of the inefficient, or tedious iteration tasks a breeze and a lot more efficient when it comes to both memory and calculation. The module includes great functions like count(), chain(), accumulate(), and so much more!
For the most part, I have seen itertools aliased as both it and its, I personally prefer its.
import itertools as its
I assume we are all familiar with the Fizz Buzz coding test. Using this simple coding test, we can see the value of utilizing the more performant iteration solutions available in itertools versus using raw Python to handle our problems. Most programmers would likely take an approach akin to this in order to solve the problem:
for num in range(1, 21):    if num % 3 == 0 and num % 5 == 0:        print('FizzBuzz')    elif num % 3 == 0:        print('Fizz')    elif num % 5 == 0:        print('Buzz')    else:        print(num)
Which of course is correct! however, using itertools, we can substantially cut down both the length of this function, and the time it will take to calculate it:
def fizz_buzz(n):    fizzes = its.cycle([""] * 2 + ["Fizz"])    buzzes = its.cycle([""] * 4 + ["Buzz"])    fizzes_buzzes = (fizz + buzz for fizz, buzz in zip(fizzes, buzzes))    result = (word or n for word, n in zip(fizzes_buzzes, its.count(1)))    for i in its.islice(result, 100):        print(i)
Unlike many of the tools on this list, the bisect module has a direct purpose. The purpose of this module is to bisect arrays, or lists in Python. This can come in handy for all sorts of different operations, however often this package is kicked to the side by a Scipy implementation under the same name. If I had to guess as to why that might be, I would say that maybe the Scipy implementation has support for Numpy matrices?
Regardless of its cultural acceptance, those working with just plain Pythonic lists will most certainly find this module very valuable. The module comes with tools to bisect normally, as well as well as to the left and right. Furthermore, the module comes with a few more interesting abilities such as insort(), which will insert and sort the values provided. Let’s import the module and see how it works with a list:
import bisect as bs
Now we can make a list and bisect it using this module:
ls = [5, 10, 15, 20, 25, 30, 35, 40, 45, 50, 55, 60, 65, 70, 75, 80, 85, 90]bs.bisect(ls, 9)
The os module is an incredibly popular Python module that is of course in the standard library. The os module comes with tools for working with file-systems, system hardware, and even the kernel of the operating system. The os module also has three functions:
stat
fstat
lstat
These are all used to either get the status of a file, or get a descriptor for said file. The stat function in the os module will return a stat_result type, which is precisely what we need in order to work with the stat module. As a result, the first step in working with this module is going to be getting a stat from the os module. Let’s start by importing the module and setting the stat type as a new variable:
import osst = os.stat("logreg.ipynb")
Now let’s look at the type of this new class. If everything went according to plan, st should be of type stat_result.
Whenever we look at the documentation for the stat module, we quickly learn that this type is not what we are actually going to be putting into our new function. We can observe this because the function stat.filemode(), for example, takes an integer, not an os.stat_result. What we will actually need is actually contained in the class we just created and is called st_mode. We can call it like so:
st.st_mode
Now let’s import stat.
import stat as stt
Now we can pass our new stat mode from our os.stat_result class into the stat.filemode() function:
stt.filemode(st.st_mode)
Scripting tasks can be incredibly difficult! This is especially true if there are a lot of concurrent services and sub-processes that might be running in the background together under systemctl or something similar. There can be a lot to a plethora of applications, and inevitably some applications are almost certain to have at least some issues along the way. A great way to mitigate a problem like this is taken straight from the Dev-ops handbook,
logging.
Logging is an easy way to keep track of several processes at once, as well as find out a bit more about what is going on behind the scenes when certain issues and exceptions are encountered. Logging provides a set of easy to use functions for simple logging. These are debug(), info(), warning(), error() and critical().
Needless to say, logging is an incredibly useful tool that all Python programmers might want to be familiar with. This is especially true of certain domains like Dev-ops, or System Administration, but this also applies to the wonderful worlds of software engineering and Data Science. There are many situations in deployment, and day-to-day computer operations where logging is going to come in handy, especially if you want to develop and debug your own applications. Now let us get our hands dirty with some Python and get to creating our own log! The first thing we are going to want to do is of course import the logging module:
import logging as lgg
Now I am going to use the basicConfig function to create a new file that our logs can be placed into:
lgg.basicConfig(filename='mylog.log', level=lgg.DEBUG)
Now I will log some messages into it!
lgg.debug('This message will be in mylog.log')lgg.info('So will this :)')
Looking at our file-system, there is a new log file we can open up that will have our text inside of it:
A cool feature of this logging implementation is that we can ignore information that we don’t want to see in our log. For example, we could set level at 20 if we did not wish to see any debug messages.
The html module is primarily known for a sub-module contained within it titled “ parser.” Given the name, it is actually pretty easy to imagine exactly what this module is for. The module can be used to easily parse HTML on a tag by tag basis. This can make it a lot easier to work with the HTML present in the text you are reading, and can even make it possible to translate HTML into a C-based frame-work with a Python API.
There are many times when a programmer could come across a large HTML page they might not want to go through, or a string received from a request that might be incredibly difficult to parse through with just string slices. If need be, you could even use the parser along with some code to actually write HTML! There are many uses for this tool, but I decided instead of putting all of the carrots into one basket I would just demonstrate the parser that you can import from html.parser.
from html.parser import HTMLParser
Now that our parser is imported, we will create a new variable and initialize an HTMLParser class:
parser = HTMLParser()
Last but not least, we will use the feed() function contained within the HTMLParser class and feed some HTML into it:
parser.feed('<html><head><title>parsedhtml</title></head>'            '<body><h1>this was parsed</h1></body></html>')
In most situations, this parser is a lot less of an HTML-parser, and much more of an HTML splitter. That is to say it is very likely that most programmers that utilize this module are going to be building a customized parser over this one.
Although this module is certainly not the most useful on this list, I do think that it is a pretty cool one that might come in handy. That being said, the webbrowser module is exactly that; a shortcut to your web-browser. With that in mind you can consider that its usage is most often to automatically pull up web links of some sort.
There are probably several different reasons that one might want to do this. I can think of applications mostly in software engineering that might apply, but there are almost certainly more useful applications for this module. Regardless of its actual usefulness to the average programmer, I think it is a pretty neat little module, so let’s try it out:
First, we can execute directly through Bash in order to open our link:
!python3 -m webbrowser -t "towardsdatascience.com"
Or we can of course navigate using Python code:
import webbrowserwebbrowser.open_new("lathe.ai")
File Transfer Protocol, or FTP, is the way we used to transfer files back when top of the line central processing units only had four cores. Regardless of its age and respective replacement (with so many better solutions,) FTP has somehow managed to stick around to some extent over the years. That being said, there are situations where you might run into actually having to use FTP.
Just because you have file access to a server does not mean that you have SSH access to a server. That in mind, without the use of SCP or SSH it would be pretty hard to connect to an open port and move your files. Fortunately, FTP can always step in when a situation ends up like this.
While I would love to demonstrate the ftplib package, I am not a big fan of the protocol to start with; and don’t really have a great means of working with it. Of course, I could probably go about setting up FTP on my server and then go back and connect to it from Python in my notebook, but the module and its documentation are relatively basic, so it is likely the code can be absorbed from just looking at the code there:
>>> from ftplib import FTP>>> ftp = FTP('ftp.us.debian.org')  # connect to host, default port>>> ftp.login()                     # user anonymous, passwd anonymous@'230 Login successful.'>>> ftp.cwd('debian')               # change into "debian" directory>>> ftp.retrlines('LIST')           # list directory contents-rw-rw-r--    1 1176     1176         1063 Jun 15 10:18 README...drwxr-sr-x    5 1176     1176         4096 Dec 19  2000 pooldrwxr-sr-x    4 1176     1176         4096 Nov 17  2008 projectdrwxr-xr-x    3 1176     1176         4096 Oct 10  2012 tools'226 Directory send OK.'>>> with open('README', 'wb') as fp:>>>     ftp.retrbinary('RETR README', fp.write)'226 Transfer complete.'>>> ftp.quit()
https://docs.python.org/3/library/ftplib.html
Universal unique identifiers, or UUIDs are an incredibly important concept in the world of computing. This is especially true of modern enterprise computing, which may sometimes have hundreds or thousands of different things to identify and work with on a daily basis. UUIDs allow for the computer to easily distinguish what exactly the data is. UUIDs are used widely across the world of modern computing in all sorts of applications such as
web development
database administration
process management
package development
and so many more.
The great thing about the uuid module is that it is also incredibly easy to use. You can retrieve identifiers for just about anything by simply using the four uuid1–5() excluding 2 methods that are provided by this module. The first and most commonly used uuid() method is most certainly uuid1(), however, many also use uuid4() because uuid1() will also tag the identifier with details on which computer created that identifier. In order to start with these functions, we will need to import them.
import uuid
Now we will just call the uuid1() function:
uuid1()
We will see that this will return a UUID. The next thing we could do is get a printout off all the different UUID functions in order to see what their output might be like. In order to use the other two functions, UUID3() and UUID4(), we will need to supply a namespace identifier. This is usually done through two different protocols, either Domain Name Servers (DNS’s), or Uniform Resource Locators (URLs), and allows the identifier to be created based on a namespace. This will create our new UUID based on the SHA-1 hash for an identifier. For example, we could set our namespace to towardsdatascience.com. For the namespace argument, we will provide a namespace object, which comes packaged in the uuid module. As for the name, we will be providing towardsdatascience.com. As you can see with the examples provided, the center of uuid1s and uuid3s usually remains consistent:
print(uuid.uuid1())print(uuid.uuid3(uuid.NAMESPACE_DNS, "towardsdatascience.com"))print(uuid.uuid4())print(uuid.uuid5(uuid.NAMESPACE_DNS, "towardsdatascience.com"))print(uuid.uuid1())print(uuid.uuid3(uuid.NAMESPACE_DNS, "towardsdatascience.com"))
Whenever you are working with accounts, and perhaps trying to make those accounts secure, it makes a lot of sense to keep track of Internet Protocol Adresses, or IP adresses. There are two versions to this protocol, and this protocol serves as the basis for all communication between computers and servers across the whole internet. Anytime your computer connects with another computer somewhere, you are likely being identified by your IP address. These addresses provide a lot more security to the internet, because it makes it a lot easier for both websites and your internet service provider to keep track of what user is doing what.
Whether that is for better or for worse is an entirely different conversation in the world of data ethics, but the fact is that ipaddresses are a tool that most server administrators and programmers in general should be familiar with. In Python, we have a standard library module for this called ipaddress. For the most part, this module is generally used to address IP networks and local IP configurations on servers. A great example of a popular application where this module is used would be Gunicorn3, a Pythonic high-performance web-server. If you would like to know more about Gunicorn3, and perhaps deploying Flask with it, I wrote an article about a year ago (actually almost two years) that I think utilizes it quite well:
towardsdatascience.com
For the most part, this package is just used to create classes off of IPs and bring them out of the string world and into the Python world, where they are their own unique type. That being said, just about the only thing to demonstrate is how to create such a class:
import ipaddressipaddress.IPv4Address('192.168.0.1')
Get text is another really cool tool I could most certainly see getting some use from the standard library. Gettext is a really cool implementation of translation into the base of Python, which I think is pretty awesome! Byfar the most popular method used in this regard is the gettext() method. The method takes a parameter that will be a string, and will be the text that we wish to translate.
import gettext as gxt
In order to actually work with this package, we will need to load language data into it. This is quite common, because it is likely that the module would be absolutely enormous without this ability. The process is actually lengthy, and I plan to cover it in a future article as it will simply take too long to explain here. However, I can explain the basic process and the advantages of using gettext before discussing the one serious issue there might be with this tool.
Gettext is used by generating a .pot file. The .pot file contains the translation information for an individual circumstance or an entire application. This file is then loaded and will load the into the set locale when used with the gettext() method. This is a great way to work with translation, because it keeps the programming and the translating completely separate.
gtxt.gettext("Hello world!")
The only significant flaw with this tool is likely its licensing. Gettext is released under GPL and other free software licenses. This is problematic because it might mean that the tool is unavailable for use commercially. This means that essentially any private use, or use at work, is infringing on the license it was distributed with. Licensing is an important thing to pay attention to in the world of software, because you can get into a lot of trouble by using software with the wrong license the wrong way. That being said, an interesting part of that licensing discussion is that gettext is being packaged with Python in this case, which is packaged under the PSF language — which is a much more liberal and free license that I would compare to the MIT license. Both of those licenses would probably insinuate that this could be used in a professional setting, however I am not a lawyer and cannot really say whether or not the licensing of this software is going to keep its use from the real world.
Probably the most valuable standard library module for Data Science on this list is the pickle module. The pickle module is used to serialize types into compressed data that can later be loaded and used once again inside of a different environment. This is incredibly useful for things like machine-learning modules, which often contain classes filled with dictionaries and numbers to determine what the model might need to do in specific circumstances.
In order to use this module, we will need to prepare a class we would like to serialize. For this example, I decided to make a class that was all about people:
class Person:    def __init__(self, feet, inches, weight):        self.height = feet * 12 + inches        self.bmi = weight / self.height ^ 2        self.weight = weight
Now I will create myself:
emmett = Person(6, 4, 160)
That’s strange...
I find it so odd that Python does not have regard for order of operations. I also think its strange that ^(float, int) does not work. I have been spoiled with Julian mathematics, I suppose. We can fix this error with a set of parenthesis:
class Person:    def __init__(self, feet, inches, weight):        self.height = feet * 12 + inches        self.bmi = weight / (self.height ^ 2)        self.weight = weight
Now let’s create our emmett object and serialize it!
import pickle as pklemmett = Person(6, 4, 160)
In order to serialize our new class into a file, we are going to use the pickle.dump() function. The best way to use this function is using the with syntax. The with block will allow us to define a new variable as an open writable file and then plug that file stream into our function as well call it:
with open("emmett.pkl", "wb") as output:    pkl.dump(emmett, output)
Now we will do the same, but instead we will load using the pickle.load() function. We will also change the write privileges to read privileges by replacing our w with an r.
with open("emmett.pkl", "rb") as inp:    emmett2 = pkl.load(inp)
Now emmett2 is identical to the emmett class, and can be moved, shared, and reused!
A programming concept that will likely change the way you work with strings the first time you use it is regular expressions. Regular expressions are cool little commands that can be placed within a string in order to tell the computer how to handle said string. The re package provides regular expression matching that is similar to that of the Perl programming language.
That being said, the re module in Python is used to detect and parse these regular expressions. This can be useful for applications that might need to interpret a lot of incoming strings from other applications. These strings might need to behave a certain way or do a particular task when coded with a regular expression. A great example of this that most programmers are likely familiar with is markdown. Markdown uses expressions in order to tell when to use certain header heights, bullet points, images, and other text features such as hyper-links.
The most commonly used function in this package is certainly going to be the compile() function. This function can be used to turn any string into a regular expression object that can then be utilized all across the package. I do not think that anyone has ever felt the need to alias this one:
import re
Let’s try that compile() function on a string:
st = "hello world, it has been a good day. \n hello world, it has been a bad day. \n"re.compile(st)
When working with types, there is always going to be a conversation on mutability. Mutability has its fair share of disadvantages and advantages. Python is a dynamically-typed language, which means that types can frequently change. Not only that, but the language also carries a lot of more iterative features, meaning that it is common for the language to not really care that much about preserving data. It is for this reason that it is going to be incredibly important to consistently create copies of your types in order to avoid a disastrous and irreversible mutation to your class’s data.
Fortunately, making sure to be safe and copy your work is relatively easy in Python. There is a standard library tool that comes with your Python installation called copy.
import copy as cp
The copy module will provide two functions, copy and deepcopy. The difference between the two is that copy creates references to child objects, whereas deepcopy recursively copies those objects. With that in mind, it might be wise to deepcopy classes that are built with classes below it, but it might make more sense to just use copy when working with simple data-types inside of a class.
emmett2 = cp.copy(emmett)emmett2 = cp.deepcopy(emmett)
Function tools is a broad toolbox of tools that usually interact with the Python compiler that can be used on functions in order to control specific ways that they act. There is a great example of this usage in the documentation that I think highlights why it is most certainly a useful package to work with:
import functools as futfrom functools import lru_cache
Now let’s take a look at one basic example where functools can be incredibly useful and really awesome! A surprisingly difficult calculation for a computer to make is the factorial calculation. Often these are calculated using a lookup table because their calculation is actually so difficult to do. This is for a reason, as factorials in and of themselves are a great way to explain recursion. We could write a recursive factorial function in Python like so:
def factorial(n):    return n * factorial(n-1) if n else 1
The only negative to this function is that it is going to continuously call itself in order to calculate the factorial. This is going to destroy performance with each use of the function. However, that being said, we can improve subsequent calls by storing the results of previous calculations in memory. The functools module allows us to do so with the cache call:
@lru_cachedef factorial(n):    return n * factorial(n-1) if n else 1
Now each time the factorial calls a calculation that has already been cached, it is instead simply going to load the number from memory! This will save our performance dramatically.
There are well over 200,000 packages for the Python programming language, and a lot of those packages are very established and venerable. However, dependencies can end up being quite difficult to handle, and often there are ways to get around using environment space by taking a look into the standard library. Additionally, there are a ton of tools inside the standard library that could certainly get jobs done and give you plenty of ideas on what to program next. In the future, hopefully some of these fantastic tools will come into use, and the information provided here on these modules will come in handy! Thank you for reading!