Profiling Apache Beam Python pipelines
By: Israel Herraiz
Tags: Google Cloud Dataflow, Profiling, Python, Apache Beam, Data

How can I interpret all that output produced by the profiler?

To parse and analyze the output of the profiler, you can now use the pstats module of Python. With that module, we can read the profiling data, aggregate and sort it, and produce a smaller profiling output file, that will make the detailed analysis faster.

With the following script you can import the downloaded data, and generate the summarized output. Here we sort by 'cumulative' , which is the total accumulated CPU time spent in that function:

After running this script, you will have a file named dataflow.prof that you can use to investigate the CPU and memory consumption of your pipeline.

That file is much smaller than the raw data you have downloaded from GCS. For instance, for this post I generated around 1.2 GB of data, and the output file had a size of 427 KB.

We can now investigate this file dataflow.prof with different tools. There are tools such as gprof2dot, tuna, or snakeviz that allow you to obtain a graph with the profiling information, and in some cases also explore the graph interactively. However, due to the large amount of irrelevant function calls included in those graphs, it is difficult to make sense of the profiling data with those graphs.

So in this post we are going to use Python to try to filter the profiling data, and find the spots that are more relevant to identify points where we could improve the performance of our pipeline.

Let's start by reading the file that we have generated, and printing some overall stats. The script is very simple:

The output will be something similar to this:

Profiling statistics for the code of our pipeline

CPU time is only available at function level. On the right side, we see the data of the function: filename, line where the function starts, and between parentheses the function name.

On the left side we can see the number of times that function has been called, the cumulative time spent in that function, and the average time (cumtime percall) spent in that function.

What are the differences between tottime and cumtime ? This difference is important and will determine how we have to analyze the output. On one side, tottime only counts the time spent in code inside the function, but it is not counting any time spent in other functions called from the code inside the function.

On the other side, cumtime is more relevant to the way we are going to analyze code. It includes all the time spent inside the function, including also any call to any other function. This will help us to find bottlenecks in our code, even if those bottlenecks are due to calls to libraries that are not our code. This is probably the kind of information we want when profiling a data pipeline. We are not interested only on algorithmic complexity, but also in any additional time incurred by calling other functions, in situations like connecting to services, parsing messages, etc. Also, if for some reason one of our functions is recursive, we want to know the total spent in the function from the first recursive call, not the individual times of each one of the recursive calls. Recursive functions can be handy but also a performance hog. With tottime we would be unable to detect those performance hogs with recursive functions; we need to use cumtime to find the places where our code is consuming most of the CPU.