Guide to Big Data Joins — Python, SQL, Pandas, Spark, Dask | by Doug Foo | Towards Data Science

There are many ways to spin a disk, filet a fish, or work with Big Data — here is a quick guide.
Kaggle Movies Database [8] — 26 million ratings for 45,000 movies, data split across 5 files:
To find the top average ratings for movies you need to join links to metadata to ratings:
SELECT m.title, avg(r.rating) FROM links l INNER JOIN to metas m ON m.imdbId=l.imdbId INNER JOIN to ratings ON r.movieId=l.movieId GROUP BY m.title HAVING count(r.rating) > 2 and avg(r.rating) > 4.5
The classic way is to load up a Database, indexing and run the SQL mentioned before (or using classic SQL below):
SELECT m.title, avg(r.rating) FROM links l, metas m, ratings r WHERE m.imdbId=l.imdbId and r.movieId=l.movieId GROUP BY m.title HAVING count(r.rating) > 2 and avg(r.rating) > 4.5
Joins in RDBMS are done in 3 major ways with some platform variants:
Nested Loops —for each row of Table A lookup the matching key in Table B. An index on B makes the lookup O(A*log B), otherwise the join is SLOW — O(A*B).Hash-Join — build a hash/map of Table B by lookup key, making the join lookup very fast — O(A*1)Merge-Sort — sort both tables and merge on a single pass, not super fast unless pre-sorted — O(A+B+A log A+B log B) → O(A log A + B log B)
Nested Loops —for each row of Table A lookup the matching key in Table B. An index on B makes the lookup O(A*log B), otherwise the join is SLOW — O(A*B).
Hash-Join — build a hash/map of Table B by lookup key, making the join lookup very fast — O(A*1)
Merge-Sort — sort both tables and merge on a single pass, not super fast unless pre-sorted — O(A+B+A log A+B log B) → O(A log A + B log B)
Using Sqlite3 [1]— creating a database, tables and loading is very easy:
Loading & querying 26m rows of data with takes ~10 min (maybe we could combine-tune the first two steps ..)
load from csv/disk — 35 sec
insert to DB — 8 min
add indexes — 30 sec
group by query — 20 sec
You can also use sqlite3 command line to test and view query execution plans as seen below. Our target query takes about 21 seconds to execute after adding a few extra indexes on the join columns.
Using a SQL DB is scalable but old school. We’ll try hipster techniques next.
We can scrap the DB overhead and write the data loads and joins directly & tediously in Python:
“merge()” is a Nested Loop join without indexes. The loop has to scan the metas and links table for every rating (26m * 50k *2). With 100k reviews, it takes 5min, so 26m reviews will take forever...
“merge_wmap()” is a Hash join — we build a map for metas and links resulting in O(n*1) performance. Joining 26m rows takes just 3 sec!
I did not implement the group-by-filter — it would be relatively fast (requires a sort-scan-combine of the 26m row results) — I estimate total time to load and process ~ 0:53
raw CSV loading to arrays — 35 sec
manual merge with indexes — 3 sec
manual group-by & filter — 15 sec (TBD ~ estimate)
Raw Python is fast but ugly. Full speed of your local PC and full control of all your bugs.
Pandas[2] is the defacto package on Python for data prep. Extremely fast and easy to use, we can do load, join and group with minimal code:
Pandas is clever. You do not need to pre-define hashes or indexes, it appears to generate what's needed on the fly to optimize joins. The biggest limit is its existence on a single machine. Processing 26m rows done in ~0:17, with less code and no external systems (DB, Cluster, etc).
loading 3 csv to DataFrames — 5 seconds
joining 3 DataFrames — 8 seconds
join, group by and filter — +4 seconds
Pandas file load is way faster than my custom py~ 35sec vs 5sec ! Goes to show don’t be a hack, use libraries. Pandas is, in theory, its single threaded/process (doesn’t seem like it on my TaskManager) thus the dataset size is limited by your PC’s memory.
Nonetheless — Pandas is the ultimate way to work on small/medium data — but we want BIG DATA!
SQL is great but limited parallelization and ability to hack with. Python and Pandas are super flexible but lack scalability. Apache Spark [5] is the defacto way to parallelize in-memory operations on big data.
Spark has an object called a DataFrame (yes another!) which is just like a Pandas DataFrame and can even load/steal data from it (though you should probably load data via HDFS or the Cloud to avoid BIG data transfer issues):
I wrote two Spark join methods. Both run in parallel. The default mode (line 15) will partition your data and shuffle (spread) it around the cluster nodes. The latter “broadcast” mode (line 18) replicates the smaller tables once and only partitions and sends the large table contents. Broadcast mode can be much faster with smaller join tables.
Spark divides work across workers nodes (JVMs — set to 8 to match my # of CPU cores) — to divide and conquer back into an aggregation. Spark code and result output below:
df.groupBy('title').agg(func.mean('rating').       alias('avg_rating'),func.count('rating').       alias('r_count')).filter('r_count >2').        filter('avg_rating > 4.5').show()
(Non-lab certified results from my laptop)
First off note the run times to join 3 datasets together:
Surprising that a raw Python solution is the fastest? Hack hack!
The final results of top movies groups (includes Spark):
Pandas is remarkable fast and efficient, up the point that you have core memory. At some point, Python/Pandas will run out of memory and crash.
Spark is a good scaling solution, albeit the cluster management can be tricky. In-memory distributed processing, partitioning jobs & data + a partitioned storage strategy (HDFS or other) is the right direction.
RDBMS are reliable but have scaling limits on moving data & processing
More on Spark in the next chapters.... Oops, I forgot Dask (native Python clusters) — maybe next time.
Above stats collected from my MSFT Surface Laptop 3 — i7/16gb/256gb SSD
[0] Full source for test code (not just gists)— DougFoo’s GitHub
[1] SQLite Python Guide — Official Python Docs
[2] Pandas Guide — 10 minute tutorial
[3] Bit older analysis SQLite vs Pandas — Wes McKiney blog
[4] Spark Joins DB Deck— DataBricks Presentation
[5] Nice Detailed Intro on Spark— TDS Article by A. Ialenti
[6] PYArrow for fast DataFrame loads in Spark — Bryan Cutler IBM
[7] Install PySpark Win in 10min — TDS Article by Uma G
[8] Movie Review Files — Kaggle Datasets