Optimizing Your Python Code. Essential tips for working with large... | by Anna Astori | Towards Data Science

While you will only occasionally get to the point where you need to run a profiler to analyze your code and find bottlenecks, it’s definitely a good idea to get into the habit of writing efficient code and spotting the places where you can improve right away.
An important thing to keep in mind when looking for ways to optimize your code is that there will most probably always be some trade-offs to accept. For example, it’s either a faster running piece of code or a simpler one. And simplicity here doesn’t mean just “code that looks less cool” (think about the famous Python “one-liners”). Simpler code means it’s going to be easier to maintain and test.
Also, every tip on how to optimize your Python code needs to be critically examined with regards to your case. There are, of course, general observations but you might also have a context in which those observations are irrelevant or even yield the opposite result. So you need to understand what’s happening “behind the scenes” and how it’ll work in your case.
However, there are situations where you definitely need to give your code a little boost and here are some of the points I found helpful when I was learning Python.
In Python, a great syntactic construct that is computationally more efficient for creating lists than a traditional loop is list comprehensions. So if you need to, say, have a binary feature vector for a list of numbers as data points where all negative numbers will be assigned 0 and the rest will be assigned 1, instead of:
>>> input_list = [1, 2, -3]>>> output_list = []>>> for x in input_list:...    if x >= 0:...        output_list.append(1)...    else:...        output_list.append(0)>>> output_list[1, 1, 0]
You can do:
>>> output_list = [1 if x >= 0 else 0 for x in input_list]>>> output_list[1, 1, 0]
You can try and compare which implementation runs faster using, for example, the timeit module.
You can also have nested list comprehensions just like nested loops but it is normally discouraged, precisely because it makes the code more difficult to read, maintain, and test.
In fact, in the previous example, if you were creating a vector with just one initialization value, instead of using inefficient for-loops and even list comprehensions, you could do something like that:
>>> my_list2 = [0] * len(my_list1)>>> my_list2[0, 0, 0]
A good example of an approach that can help shave off a significant amount of the runtime complexity but requires a lot of careful thought in terms of trade-offs is function calls. While you do want the nice abstraction, extensibility, and re-usability that functions provide, you might not want to have a function for every single thing, because function calls are quite expensive in Python (if you’re interested, there are interesting observations on that in this article). So sometimes, you might want to sacrifice, for example, writing a getter and/or a setter. On the other hand, fewer functions make the code less testable. So the final decision really depends on the specifics of your application.
Another tip related to functions is opting for built-in functions like max(), sum(), map(), reduce(), etc. rather than carrying out those computations yourself — they are usually written in C and will run faster. Also, if you use a built-in function — it’s less code you will have to write tests for yourself. So, for example, if you wanted a set consisting of all absolute values of input_list, you could do something like that:
>>> output_set = set(map(abs, input_list))>>> output_set{1, 2, 3}
If you’re working with textual data, for string concatenation, instead of += :
>>> sentence_list = ['This ', 'is ', 'a ', 'sentence.']>>> sentence = ''>>> for i in sentence_list:...     sentence += i>>> sentence'This is a sentence.'
use str.join():
>>> sentence = ''.join(sentence_list)>>> sentence'This is a sentence.'
With +=, Python allocates memory for each intermediate string, and only once if str.join() is used.
Use operator.itemgetter for sorting. If you have a list of tuples of, for example, first and last names like this:
>>> my_tuples =[('abbie', 'smith'), ('jane', 'adams'), ('adam', 'johnson')]
Default sorting would have returned this:
>>> sorted(my_tuples)[('abbie', 'smith'), ('adam', 'johnson'), ('jane', 'adams')]
If you want it to be sorted by last names instead of first names, you can do it like so:
>>> sorted(my_tuples, key=operator.itemgetter(1))[('jane', 'adams'), ('adam', 'johnson'), ('abbie', 'smith')]
If you have an object and you are using some of its properties, assign them to local variables first:
rectangle_height = rectangle.heightrectangle_width = rectangle.width
So if later in your code you’re computing, for instance, its surface, you will do:
surface = rectangle_height * rectangle_width
And if later you also compute its perimeter, you will re-use the same variables:
perimeter = 2 * rectangle_height + 2 * rectangle_width
You can use the timeit module again to verify that it saves you the lookup time for each reference to the rectangle object and its properties.
For the same reasons, using globals is often discouraged — you don’t want to be wasting time looking up, first, the global variable itself, and then each of its properties you may be referencing.
This can also apply to function references. If you’re processing, for example, a sequence of data points and you’re appending each resulting item to a list, you can do the following:
push_item = my_list.append
and then apply that to each resulting item:
push_item(item)
In addition to general properties of each typical data structure, for example, the complexity of retrieving an item from a linked list, it’s good to know how Python data structures are implemented and where you can save some CPU or memory there. For example, if you’re looking up a key in a dictionary, you don’t even need to reference dict.keys() which makes it a little slower in Python 3. You can simply do:
>>> if k in dict:...    do_something(k)
In Python 2, dict.keys() even used to create an extra list of the keys!
At the same time, don’t forget to take a look at the bigger picture. If you had a list of items to process and you know that a lookup in a set is O(1) vs O(n) in a list, you might be tempted to turn your list into a set:
>>> my_set = set(my_list)
However, if you’re only looking up one item of that list, you might actually make things worse if you turn your list into a set first. Behind the scenes, Python will iterate over your entire list and add each item to a newly-created set. The overhead of creating a set will make you lose the advantage of looking up in a set then.
On the other hand, if you want to efficiently remove duplicates from a list, casting it to set can be a good option (although there are also other options for that in Python that may work better for your case).
So, as I mentioned before, there are general observations but you need to examine them carefully to understand how they’ll work in your case.
There is a fair amount of more advanced approaches to managing your code in order to make it more efficient and execute faster — from parallelism or concurrency to different tricks like designing smaller objects so that they fit in the cache layer of the heap memory rather than the main one. For some tasks, you may be able to use libraries that are actually designed to optimize those kinds of tasks. However, the ones above are the first things I had to start paying attention to when I learned Python. What tricks and approaches are you using?
If you’re working in Python in Data Science, feel free to check out my articles on new features in Python 3.9 and concurrency in Python!