How to Profile Your Code in Python
By: Ehsan Khodabandeh
Tags: Data Science, Programming, Python

How to Profile Your Code in Python

Finding bottlenecks and optimizing performance using cProfile

Photo by Anthony Maggio from Pexels

If you’ve ever written a line of code (or even tens of thousands of lines), you’ve surely wondered “Why does my code take so long to run?” Answering that question isn’t always simple, but it can be easier if you search for answers the right way.

Perhaps you trust your knowledge of the problem at hand, using your subject matter expertise to check certain snippets first. Maybe you time a few different modules/classes/functions to see where most of the execution time is spent. Better yet, you can profile your code to get more information about the relative time spent in different functions and sub-functions. Whatever your process is, this blog might teach you some ways to get to the answer more quickly.

In this post, I’ll start off by showing you a basic profiling method. I’ll add more and more features and flavor to it, ending with a good, solid profiling decorator. For those in a hurry (or who want to refer back to this material), head over to this GitHub repository where you can find the profile decorator and an example.

Time It!

To profile your code, you’re gonna need to know how to time it. To do so, you can use something as simple as this:

from time import time start = time()

# your script here

end = time()

print(f'It took {end - start} seconds!')

To easily time several functions/snippets (or if you just prefer to use a cleaner and more pythonic approach), you can convert the above into a timer decorator (discussed with an example here).

Using a timer on any function shows us the run time of that piece in isolation. For this approach to be useful in finding the bottleneck, we need more information. At least one of the following two conditions should be true in order to profile your code effectively:

We should know the total run time of the program to have a better picture of the relative run time of our desired function/section. For example, if a piece of code takes 5 minutes to execute, is that 10%, 40%, or 90% of the total run time? We should know enough about the problem at hand or the run time of other parts of the program to confidently label a given piece of code as the bottleneck. Even if a function takes 10 minutes to run (let’s assume 10 minutes is relatively high), we should only worry about its inefficiency if we’re confident that there’s no other part that takes longer. As Donald Knuth famously said:

Premature optimization is the root of all evil.

A profiler package like cProfile helps us find the bottlenecks in our code by satisfying both of these conditions.

How to Use cProfile

Basic Usage

The most basic way of profiling with cProfile is using the run() function. All you need to do is to pass what you want to profile as a string statement to run() . This is an example of what the report looks like (truncated for brevity):

>>> import cProfile

>>> import pandas as pd >>> cProfile.run("pd.Series(list('ABCDEFG'))") 258 function calls (256 primitive calls) in 0.001 seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function)

4 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:997(_handle_fromlist)

1 0.000 0.000 0.000 0.000 <string>:1(<module>)

1 0.000 0.000 0.000 0.000 _dtype.py:319(_name_get)

....

11/9 0.000 0.000 0.000 0.000 {built-in method builtins.len}

1 0.000 0.000 0.000 0.000 {built-in method numpy.array}

1 0.000 0.000 0.000 0.000 {built-in method numpy.empty}

....

The first line indicates that 258 calls were monitored, out of which 256 were primitive (a primitive call is one that was not induced via recursion). The next line, Ordered by: standard name , indicates that the report is sorted based on the standard name, which is the text in the filename:lineno(function) column. The line after that is the column headings:

ncalls : is the number of calls made. When there are two numbers (like 11/9 above), the function recurred. The first value is the total number of calls and the second value is the number of primitive or non-recursive calls.

tottime : is the total time spent in the given function (excluding time made in calls to sub-functions).

percall : is the quotient of tottime divided by ncalls .

cumtime : is the cumulative time spent in this and all subfunctions. This figure is accurate even for recursive functions.

percall : is the quotient of cumtime divided by primitive calls.

filename:lineno(function) : provides the respective data of each function.

The run() function can accept two more arguments: a filename to write the results to a file instead of the stdout, and a sort argument that specifies how the output should be sorted. You can check the documentation to learn more about valid sort values. Some of the common ones are 'cumulative' (for cumulative time), 'time' (for total time), and 'calls' (for number of calls).