10 Surprisingly Useful Base Python Functions | by Emmett Boudreau | Towards Data Science

Those that work in Python often consider that maybe Python isn’t the star of the show here — but instead its brilliant ecosystem. Often whenever we are working with Python applications, we are working with a lot of dependencies. Python on its own isn’t particularly good at anything, but it is instead its wonderful packages that bring the functionality to the base.
Whenever I have this discussion, I often think of NumPy as an example. While matrix math might be possible with vanilla Python, utilizing NumPy makes it substantially better — and even faster. That being said, while often it seems Python developers do rely a lot on external tools to process certain datatypes and work mathematically especially, the Python language does have some really cool tricks up its sleeve that do make it appeal to a scientific programming audience. Using these tricks you can keep your dependencies down, free up your time, and generally make programming a lot easier in the Python programming language! Next time you are thinking of installing something with Pip, you’ll be able to also consider the options available in the standard library!
I wrote an article a while back on why I think Python’s lambda makes it a far more viable programming language for statistical computing. Furthermore, since Python disobeys some more restrictive object-oriented rules, it is able to act very functionally and declarative — bringing the conversation to lambda, which can be used to apply mathematical operations to any datatype by using expression evaluation rather than entire functions. This can be used for things that are more simple to program, and likely don’t need an entire function to their name. You could even define these in a global scope, since it is Python, and take Python to another scientific level by leveraging functional-like syntax and methodology in a language that still has class structure.
While this concept might seem like a small bump to some, utilizing it saves time, memory, and can make your code look a lot more concise. Along with lambda, since the return is a Pythonic expression, we also get the ability to use methods like apply() to apply expressions quickly across entire subsets of data. Needless to say, for a Data Scientist these operations come plenty in handy, and being able to do them in one line very effectively makes our global-everything style of programming a lot easier in Python! The syntax goes like this; we start by setting a value equal to the return of lambda followed by the variable we would like to have provided as a positional argument. Then we perform our operation using that provided argument as a variable, for example the mean:
mean = lambda x : sum(x) / len(x)
Now we could call it just as we would any method in the Python programming language:
x = [5, 10, 15, 20]print(mean(x))
If you would like to read more about Lambda, why I love it, and how to use it, I wrote another article on it as well. The great thing about this article is that it goes in-depth and shows various tricks that Lambda can be used with in Python. Check it out!:
towardsdatascience.com
One of the most underrated tools in the Pythonic arsenal is the Shutil module. Like the other modules on this list, this is included in the standard library and can be imported as you usually would any module in the language:
import shutil
So what does shutil do? The shutil module is a high-level interface for the Python programming language for the file-system on your operating system. While these calls are often made using the os module, it is the shutil module that is lesser-known and I think should be used more often. How often have you needed to quickly move a file from directory to directory with a script and had to do a whole surplus of tedious work just to get it done?
Shutil solves these classic problems with files and allocation tables by making a high-level solution to a common problem. Even compared to many other high-level, and sometimes dramatically declarative programming languages, this solution blows them out of the water. For all of your dev-ops implementations, this is certainly a key to save a lot of time and make file operations far faster than they would be otherwise. Below are some examples of the high-level calls that the shutil module provides for some basic file operations:
import shutilshutil.copyfile('mydatabase.db', 'archive.db')shutil.move('/src/High.py', '/packages/High')
While the glob module might not be as cool as shutil, or nearly as useful for programming in a general sense as Lambda, the module is still pretty useful for certain situations. The glob module is used to search directories for wildcards. This means that it can be used to aggregate data about files and file-extensions stored on your machine. Of course, this module is imported precisely how you might expect it to be:
import glob
In all honesty, I am not sure if this module contains more than a single function, but that function is quite commonly used for the module’s exact purpose. This function is the glob() function. This function will take any typical file searches. This includes file-searches with Unix syntax, i.e. *, /, etc. Check it out:
glob.glob('*.ipynb')
This returns a list of file names that correspond to the files found by the glob’s search. This could also be used for aggregating data, or it could be used for working with files that store data!
The argparse module provides a more robust and in-depth method to parse command-line arguments. Many dev-ops tools utilize this concept so that they can be interacted with from the Unix command line. This is useful for many dev-ops tools that are commonly used by Python developers and Data Scientists. A great example of this would be the Python module Gunicorn, which can process passed command-line arguments for workers and various other parameters. Just as before, this module must be imported to be used:
import argparse
In order to use this module, we will build a new type which will be an argument parser:
parser = argparse.ArgumentParser(prog = 'top',description = 'Show top lines from the file')
Now we add arguments to our parser using the add an argument to this new reader. For this case, we will create an argument which can be passed to determine the number of lines we want printed from each file:
parser.add_argument('-l', '--lines', type=int, default=10)
I added a few key-word arguments, one of which will provide the datatype that we are expecting to be passed for this argument, and the other is a default value for when the file is called without this argument in place. Now we can get the arguments out by calling the parse_args() function on our new argument parser type:
args = parser.parse_args()
Now we can call this Python file for compilation while also providing the necessary parameters easily from Bash.
python top.py --lines=5 examplefile.txt
Needless to say, this could certainly come in handy. Some notable situations where I have used this module is when working with Crontab, which can run scripts at specific Unix timestamps. Furthermore, this script could also be used for supervisors, which run Bash commands without a user being present as a worker. If you’d like to learn more about crontab, I have a whole article talking about why I love it here:
towardsdatascience.com
Furthermore, if you’d like to learn more about some of the other applications that I use to deploy Pythonic endpoints and websites, I have a whole article detailing an A-Z deployment with a self-training machine-learning model you can check out here:
medium.com
Of all of the tools I have reviewed today so far, the re module is by-far the most critically underrated. The “ R” and “ E” in re is short for regular expressions. The re module is used to parse strings with regular expressions and provide more options for working with strings in Python. How many times have you ran into making algorithmic decisions based on functions that the string class has like str.split()? No more of that because regular expressions are far simpler, and a lot easier to use!
import re
The re module, unlike some of the others on this list, provides a lot of very useful functions; particularly for working with copious amounts of data as Scientists typically do. That being said, two examples that will get you started are the sub() and findall() functions.
import rere.findall(r'\bf[a-z]*', 'which foot or hand fell fastest')['foot', 'fell', 'fastest']re.sub(r'(\b[a-z]+) \1', r'\1', 'cat in the the hat')'cat in the hat'
Needless to say, this could be very useful for working with data that is often encountered in this world.
Ok — hear me out. Math might not be the greatest module in the history of the standard library, but it does frequently come in handy. This of course is even more true for scientific computing. The math module gives way to access to everything from sin and cos all the way to logarithms. All of these things can come in handy when working with algorithms, needless to say, and many packages do depend on this basic module to get a lot of things done.
import math
The module can certainly save some time by making mathematical operations easily available without dependencies. For this example, I will be demonstrating the log() function, but you could most certainly look further into this module and find a whole world of mathematical capabilities!
import mathmath.log(1024, 2)
Another package that is almost certainly a no-brainer for scientific computing is the statistics module. This module provides some elementary statistics that might not go as in-depth as something like SciPy, but can still suffice quite frequently for data analysis. While inferential and Bayesian statistics might not be a key focus of this package, mathematical values can such as the mean or standard deviation can tell a lot about the data that you might be working with. I frequently alias this module as st, but I have also seen it aliased as stc or sts, not to be confused with the common alias scs for Scipy.stats.
import statistics as st
There are plenty of useful functions provided by this module that are certainly worth checking out! The great thing about this package is that it will always be there, whether you need it or not, and it won’t require any dependencies. Let’s look at some basic general-purpose statistical operations that are easily provided with this package:
import statistics as stst.mean(data)st.median(data)st.variance(data)
While many of these packages might not be all that well-known, the urllib module is most certainly an exception to that rule. This module can be used to handle many common web-related tasks such as requests. Let’s import it!
import urllib
Often requests are something that Data Scientists do with a dependency. A common too that fits this description is Flask’s requests module. This module is another commonly used solution for requests, and comes with more functionality than urllib, such as the ability to request query arguments from the http request. That being said, it can be convenient to use the standard library and worry a lot less about dependencies so long as the project doesn’t require these extra features.
from urllib.request import urlopendata = nullwith urlopen('http://example_url/') as response: data = response
Although the urlopen function is certainly a great example for a common usage of urllib, there is a lot of functionality to be had in this relatively small package, and it is definitely a tool that is worth learning in my opinion.
Another great example of a tool that is quite common amongst scientific computing is a date-time type. It is very common that data is going to have timestamps, sometimes timestamps are even the predictive feature that is going to be training our model. That being said, working with dates and times might be a relatively important concept for a Data Scientist. We can import the date-time module exactly how you might expect; it is very common to see this module aliased as dt, so for this example I am going to be doing precisely that:
import datetime as dt
Now we can construct date-time types and work with the typical date-time syntax with properties like year, month, and day. This is incredibly useful for re-formatting, parsing, and working with individual sections of dates within your data. Let’s take a look at some basic functionality from this package:
import datetime as dtnow = dt.date.today()print(now.year)print(now.month)
The final contender on this list is the zlib module. The zlib module is an all-around solution for compressing data using the Python programming language. This can come in handy for package distribution, especially if package distribution is entirely automated. One great thing about Python is that it can be used as a fantastic automation tool. That being said, being able to automatically compress a tarball and then even send it somewhere else is a pretty valuable asset.
import zlib
Within the zlib module, the most important functions are probably compress() and decompress(). Using these functions, we can actually see the compression of datatypes before our eyes using this code:
h = " Hello, it is me, you're friend Emmett!"print(len(h))t = zlib.compress(h)print(len(t))z = decompress(t)print(len(z))
Although often in Pythonic programming it can seem like there is a high reliance on dependencies, the Python standard library also has some really neat tools in it! A lot of these tools can quickly be used to save a substantial amount of time by making a high-level interface for complex tasks that is packaged with the programming language itself. This can be used to minimize dependent packages, virtual environments, and general clutter around a server. Furthermore, there is no need to pip install any packages, so a lot of time is saved with this initiative.
These are only some of my favorite modules that are available in the standard library, however there are plenty more! While these modules will certainly come in handy, I am certainly sure that there are plenty of other packages available that will make a lot of programming operations a lot easier than they would be otherwise.