Understanding Spark As If You Had Designed It
By: Felipe Melo
Tags: Distributed Systems, Software Development, Spark, Software Architecture, Big Data

Understanding Spark As If You Had Designed It

From a simple function to a resilient and distributed framework.

Why caring about Spark?

Among the current frameworks available on the data space, just a few have achieved the status that Spark has in terms of adoption and delivery. The frameworks has emerged as one of the clear winners specially on the Data Engineering side of the landscape.

If you are reading this article, it means that you already understand the reasons behind the previous paragraph, so we are jumping directly into the main subject.

Why caring about Spark internals?

Someone might argue that we do not have to understand how an engine works in order to drive a car, which is true. However, one might also argue that understanding the engine makes you a better pilot, as you would be able to understand capabilities, limitations and eventual issues with the whole vehicle.

Following the same rationale, you don’t have to understand Spark internals to make use of its APIs. However, if you do, a lot of pain from poor performance to cryptic bugs will be alleviated. Also, you will grasp concepts pervasive across the whole field of distributed systems.

The approach

In my understanding there are two ways to learn something: episteme and techne. The former is related to formal knowledge acquisition, through books, structured courses and so on. It is more focused on the what. The latter is related to the craft, the “learning by doing” which is more focused on the how. This is the path we are taking here.

We are going to start from a simple problem that every beginner programmer could solve and evolve it to justify the architectural design of Spark. We will also understand HDFS (partially known as Hadoop) along the way, as it is a platform that plays very well with Spark.

We are going to be language-agnostic, so all the code will actually be pseudo-code.

The problem

You have been hired and assigned a simple task: count how many even numbers you have in an array.

You will read this array from a CSV file stored in your local file system. Without thinking too much you’d probably end up with the following chunk:

New requirement 1

Your clients are delighted with the immense success of the previous solution, and people now think they can throw every problem onto you, so they ask you to also calculate the average of those even numbers.

You certainly know about the SOLID principles and specially the Single Responsibility one, which says that a class or method should only have one reason to change. However, you decide to break the rules and just implemented as below:

New requirement 2

Since you were so quick, people came out with yet another requirement: also return the sum of all even numbers.

At this time you start considering not only the SOLID principles but also the way things are going. You know that usually, when something happens once it does not mean it is going to happen twice, but if it happens twice the third occurrence is right around the corner. So you start considering implementing something that is easier to extend, and you remember about the concept of encapsulation from Object Oriented Programming.

Also, it is possible that if you capture the proper abstractions, you might not even have to change your implementation when another requirement comes.

A set of abstractions to rule them all

You start by considering that if they asked you to count even numbers, it is very likely that they will further ask you about the odd ones, or those below or above a value, or those within a range, etc. So even being an expert in the application of YAGNI (You Ain’t Gonna Need It), you decide to implement something that would support all of those cases.

You conclude that in the end, all those operations are related to filtering values from the array, so instead of coding every possible filter, you decide to provide a filter function that takes the filter conditions.

Also, in order to simplify the design, you decide to change the state of the object every time you have an operation called on it.

Up to a new challenge

You did it. Now, you are not only implementing all the requirements but also getting rid of new ones involving filtering values from the array. If instead of even numbers your clients now want odd ones, the only thing they have to do is to pass a new condition to the filter method and they are done. Amazing! But that new requirement you were waiting for has just arrived: they now need you to process a 3 TB array.

You consider giving up. Your own hard disk is only 500 GB big, so you’d need 6 machines like yours entirely dedicated to store that file just in order to start. But your clients like you and are also persuasive, and after a well deserved raise, they also promised to provide you with not 6, but 30 new machines to solve the problem.

Divide

Having access to 30 new machines, you start to consider how to approach the problem. A single machine will not contain the whole file, so you will have to slice it into smaller chunks that could fit each new hard disk. Also, you consider that since you have enough resources, you can also store the same slice in more than one machine, as a backup. Maybe two copies per slice, which means you’d be able to find that slice in three different places.

You format the hard disks and start copying the file, and in the process, you consider it is a good idea to save all slices in the same canonical parent folder in every machine, and also to prefix each of them with an identifier, which is related to which section of the bigger file it belongs to. You also think it is an equally good idea to have another directory in at least two machines containing metadata about what are the directories containing the slices and which machines contain the backups for each slice id.

Since some of your machines will contain only data and some will contain only metadata giving directions and names to things, you decide to call the former data machines and the latter name machine. But since you’re actually creating a network, calling the machines nodes is more appropriate, so you have the data machines named data nodes and the metadata ones called name nodes.

Still in the process of giving things names, you realize that slice is more associated to cakes and cheese than it is to data chunks. You are feeling quite inspired and creative, so you decide to give these slices a much better name: partitions. So, whatever your program end up becoming, it will process the whole file divided into partitions.

After all those naming and decisions, you have something like this:

Your very first distributed file system

Conquer

Now you have your file divided into partitions across a set of nodes (which we will creatively call cluster for now on), with backups and a metadata to help your program to find every partition and its backups. Since it does not make any sense to move the partitions, the question then becomes: to execute the same piece of code in every machine and get one single result?

Should you send the whole program to every machine every time you need to run it? Or should you have some pieces of it already available there so that you only have to send the section written by your clients? The latter sounds better, so you go with it. In this course of action, the number one requirement is to have your ArrayOperator class available in every machine and only send the section specified by the main method.

You also want to run your code as close to the data as possible, so your data nodes will have to also run your program. From this perspective, the nodes not only store data but also perform real work, so you decide to call them workers.

Some sections of your code could also run in parallel. For example, for the program above, you can execute average(), sum() and size() in parallel as they are independent from one another. To allow that, your workers will need to support independent lines of execution, so you decide to convert each worker into some kind of daemon that will spawn new processes that will execute tasks independently (in the meantime, you realize that task is a name generic enough to refer to each unit that can be executed independently). And since you’re still inspired, you decide to creatively call the processes that will execute the tasks executors.

Now all you have to do is to design your main method — which has access to your client code — to make it drive the separation of your clients’ code into the tasks that will compose the job, then ask the name node which data nodes contain each partitions of the file, then send the tasks in parallel to the worker machines, which will be prepared to launch executors that will execute the tasks and return the result. Since this piece of code will be driving the whole thing, you, still blessed by the same creativity, decide to call it driver.

Your driver also needs to figure out how to put all the results together. In this case, it needs to add up all the summations received from each worker. But considering the strides made so far, it is gonna be a piece of cake.

In summary, your driver will be coordinating the tasks that will get the job done. And here is your imagination again. Which better name to describe a set of tasks than job?

Your beautiful piece of engineering

Made to break

After some overnights, you finally have all pieces running together. What an impressive feat! You test it and everything works as expected. You’re anxious for a demo, which your clients, after the considerable investment, are equally keen to see.

You start the demo by praising yourself, which is due, and then move on to explain the architecture. Your clients get even more excited. You run the program. Everything breaks apart because five of your machines went offline, two from kernel panic, two from hard disks failures and one because of an untested feature that ended up in a bug. Everyone starts to cry, except you. Your clients lose faith, but your confidence is rock solid. You actually praise yourself again because you already have everything figured out. Those backups are not there out of coincidence.

You promise a new demo in one week. Your clients leave the session quite grumpy and somewhat sad, but you keep it all together.

“Less is less, not more”, Backups circa 2020

You had it right. Since every partition contains two other copies and you have 28 machines (remember, you reserved 2 for the name node), you’d be very unlucky if the failure of 5 machines brought your whole cluster down.

But how to take advantage of redundancy? One thing you’re sure about is that it should start on the driver side, since this is the piece that is communicating with all nodes. If you have a failed node, it will be first noticed by the driver. Since the driver is already in touch with the name node to find the locations of the partitions when the job starts, maybe it can also ask the name node about the locations of all the copies that were sitting in the failed worker/data node. With that information, it could just resend the tasks to be executed on the copies and you’d be done!

With the previous approach, you’d have the distributed processing of distributed data in a resilient way.

You go for it.

A fresh start

You call your clients and ask for a new demo. They pretend to still be frustrated but can barely hide their excitation. They come to see you and enter the place saying some jokes about the last time. You only catch “blue screen” but doesn’t quite care about the punchlines.

Before starting, you do something shocking: you ask them to randomly shutdown two workers/data nodes. They look surprised but get in the mood (it was fun to see them trying to outsmart you by randomly choosing machines with a treacherous smile).

With two nodes less, you start the demo, which works like a charm. They cry, but this time the tears are different. They cheer you up, apologize for not believing, offer you a new raise and of course, bring a new requirement: instead of numbers, the array will now contain objects with multiple properties. More specifically, the records will contain names, ages and salaries, and they want to know what is the average age and maximum salary of people called Felipe. And they also want to save the result so that it can be accessed later without reprocessing.

You’re not surprised at all.

The cherry on top of the cake

At this time you don’t have to think much. You have been playing with abstractions all the time, so now it is just a matter of moving one more level up.

You depart from your previous design and change it like this:

With that new design, you can now process any kind of record (this is why you changed its name to GeneralOperator).

This is truly amazing! Think about it. You have a system that can read, write and process any kind of datasets in a distributed and resilient way. Speaking even more freely, you can claim you have a framework that supports the processing of resilient and distributed datasets of any kind.

You feel the power that lies in your hands, but you think that the core of your magic, the GeneralOperator does not have a catchy enough name. Or at least it is not very self-explanatory. You don’t have better ideas though, so you just decide to call it Resilient and Distributed Datasets Reader, Writer and Processor. But that is too big. So maybe an acronym, like RDDRWP? Ouch, even worse. And what about RDD only? Easy to pronounce and sounding in case someone asks you to translate. Sounds good enough, you’re done.

TLDR;

Here is what you have done:

1. You have devised an infrastructure to store replicated data partitions in a distributed fashion composed of data nodes that hold data and name nodes that contain metadata about it (doesn’t this pair deserve a name of its own? What about HDFS?)

2. You created a structure called Resilient and Distributed Datasets (RDD for short) that can read, write and process data stored in a Hadoop cluster.

3. You have architected an infrastructure to execute tasks in parallel on the distributed partitions through Workers that control the execution on a given node and Executors which actually execute the tasks.

4. You devised a driver application that breaks a job provided by a client into multiple tasks, talks to name nodes to find out where partitions are and sends the tasks to remote workers.

Man, you rock! But doesn’t your creation deserve a nice name? It’s been so many ideas, one spark after another. Yeah, Spark! That sounds like a name!

You can market it like this

Scaling Up

This stuff you created certainly has a ton of value, but maybe it has a somewhat steep learning curve. On the other hand, the language of choice to crunch data has been, for a long time (maybe too long), Structured Query Language, or SQL. What about bringing that kind of capability into your Spark?

Let’s chat with the clients.

IMPORTANT

The above is a very simplified view of Spark components and its main intention was to offer a general grasp of Spark’s architecture. Elements related to Catalyst, scheduling, transformation types, shuffling, plans, resource allocation, specialised API methods and others were intentionally left out in order to make the text simpler. They will be approached in further writings.

FURTHER READINGS

On Spark: https://data-flair.training/blogs/apache-spark-ecosystem-components/

On RDDs: https://spark.apache.org/docs/latest/rdd-programming-guide.html

On OOP: https://en.wikipedia.org/wiki/Object-oriented_programming

On S.O.L.I.D principles: https://scotch.io/bar-talk/s-o-l-i-d-the-first-five-principles-of-object-oriented-design

On YAGNI: https://martinfowler.com/bliki/Yagni.html