Is Python Really a Bottleneck?. Full disclosure — I’m currently working…
Tags: Software Development, Data Science, Data Engineering, Software Engineering, Python

What are the true bottlenecks in data processing?

Based on my own work, I usually experienced bottlenecks not in the language itself but rather in the external resources. To be more concrete, let’s look at several examples.

Writes to relational databases

When processing data in the ETL-fashion, we need to load this data in the end to some centralized place. While we could leverage multithreading in Python to write data to some relational database faster (by using more threads), the chances are that the increase in the number of parallel writes could max out the CPU capacity of that database.

In fact, this happened to me once when I was using multithreading to speed up the writes to an RDS Aurora database on AWS. I then noticed that the CPU utilization for the writer node went up so high that I had to deliberately make my code slower by using fewer threads to ensure that I wouldn’t break the database instance.

This means that Python has mechanisms to parallelize and speed up many operations, but your relational database (limited by the number of CPU cores) has its limits that will unlikely be solved just by using a faster programming language.

Making calls to external APIs

Working with external REST APIs from which you may want to extract data for your data analytics needs is another example where the language itself doesn’t seem to be a bottleneck. While I could speed up the data extraction by leveraging parallelism, this could be in vain because many external APIs limit the number of requests we can make within a specific time period. Thus, you may often find yourself making your script deliberately slower to ensure that you don’t exceed the API’s request limits:

time.sleep(10)

Working with Big Data

From my experience working with vast datasets, you can’t load really “big data” into your laptop’s memory regardless of which language you use. For such use cases, you will likely need to leverage distributed processing frameworks such as Dask, Spark, Ray, etc. There is a limit to how much data you can process when using a single server instance or your laptop.

If you want to offload the actual data processing to a cluster of compute nodes, possibly even making use of GPU instances that can further speed up compute, Python happens to have a large ecosystem of frameworks that make this task easy:

Do you want to speed up compute for data science by leveraging GPUs? Use Pytorch, Tensorflow, Ray, or Rapids (even with SQL — BlazingSQL)

Use Pytorch, Tensorflow, Ray, or Rapids (even with SQL — BlazingSQL) Do you want to speed up your Python code to process Big Data? Use Spark (or Databricks), Dask, or Prefect (that abstracts away Dask under the hood)

Use Spark (or Databricks), Dask, or Prefect (that abstracts away Dask under the hood) Do you want to speed up your data processing for analytics? Use fast specialized in-memory columnar databases that ensure high-speed processing just by using SQL queries.

And if you need to orchestrate and monitor data processing that occurs on a cluster of compute nodes, there are several workflow management platforms, written in Python, that will speed up the development and improve maintenance of your data pipelines, such as Apache Airflow, Prefect, or Dagster. If you want to learn more about those, have a look at my previous articles.

As a side note, I can imagine that some people complaining about Python don’t leverage it to its full capacity or may not be using proper data structures for the problem at hand.

To summarize, if you need to process large amounts of data quickly, you will likely need more compute resources rather than a faster programming language, and there are Python libraries that make it easy to distribute work across hundreds of nodes.