What is Apache Spark and distributed computing?
By: Michelangiolo Mazzeschi
Tags: Big Data Analytics, Apache Spark, Spark Cluster, Advanced Analytics, Distributed Computing

What is Apache Spark and distributed computing?

In this article, I’m going to talk about one of the most useful and prominent tools or big data analytics called Apache Spark. I have been using this tool for about 3–4 months and just a few days ago I have been taking the test to pass the certification for associate Spark developers using Python. The more I am becoming comfortable with this tool, the more I am understanding Spark use cases and their potential as big data software. In this article, I’m not going to show you some lines of code. It would require too much complexity. Instead, I’m going to explain to you what is this software, what are its advantages, and why as data scientists, you should start using it.

What is Apache Spark?

A spark is software able to perform data analytics on big data. It is specialized in stream processing, meaning that it can process data in real-time. is one of the most used software among the top Fortune 500 companies. The reason why Spark is so powerful is that it doesn’t run on a single computer, but it runs on a cluster (a network of computers). This is called distributed computing, and I will explain in detail what it means.

Apache Spark logo

Spark has been developed by a community of developers called Apache Software Foundation. This community is dedicated to developing a set of software for several applications in the software industry. Some of these solutions are used on Big Data, specifically Spark and Hadoop.

What do we mean by Big Data?

Before explaining in detail what is Spark, you will need to understand what we do mean by big data. Using the term big data, we identify a kind of data that requires a huge amount of computing power and also time to analyze. That is why performing analysis with big data is always problematic. You first need an expert who is able to work with big data, and second, you need a huge amount of computing power, which is very expensive and can be rent on the cloud. Only a few experts know how to perform decent analysis using big data, because you can only acquire the skill by experience and hard study, not only by experimenting. This skill is becoming more and more valuable.

Distributed Computing

In order to understand how Spark works, you will first need to have a grasp of distributed computing.

Horizontal Scaling

The most common way to perform any kind of data analytics is by using your own machine. If the data is small this won’t cause a problem, but when the data is very big, it may take too much time because of the limited processing power of your hardware. Even with the current technology, for example, a data set of a few GB will take a few hours to process when creating a machine learning model. But what if you have 100 gigabytes of files? The pc would simply require too many hours, days, or even months to finish. It is not doable. One of the possible solutions is scaling horizontally, this means increasing the GPU power to perform the task faster. Nowadays you can rent the GPU on the Internet thanks to cloud providers. This technology is called cloud computing and it’s still one of the best options to analyze big data.

Using a cluster

However, simply increasing the computing power doesn’t give you any guarantee that the way the analysis is executed will be optimal. Meaning that there might be the chance that some of the machines you have rented will waste a lot of processing power. In a few words, the process of performing analytics could be optimized. That is where distributed computing comes useful. Distributed computing does not use a simple computer or a collection of GPUs, but uses several different machines and their corresponding GPUs and CPUs. To perform the analysis the data is sliced into different partitions so that it can be used in parallel by all the machines in the cluster, with the objective of maximizing the efficiency of the process. This means that the execution of the program will be as efficient as possible, saving as much computing power as possible. The creation of a cluster is something complex and required complex software. Spark is one of the possible options.

Spark cluster

Now that you know how distributed computing works, we can talk more in detail about the Apache Spark Library. This library is open-source, meaning that you do not need to pay to use it. Simply put, this library allows you to implement a cluster, which could be a network of machines existing in the real world or even simulated in your own computer (as you can imagine, for experimentation purposes), and work on the data sliced in different partitions. By using Spark, you can manage your program in a way that is in line with the requirements of distributed computing, and so that it can run as efficiently as possible. As you might have already understood, whenever you have a huge amount of data, this software becomes incredibly useful, because every operation will allow you to save precious time and even processing power.

An overview of the Spark architecture

Spark architecture outlines this process: the Driver program splits the data into partitions which are then sent and processed in different Worker nodes, each one containing one or multiple executors that perform the calculation.

Databricks

One of the best solutions that are able to implement Spark is called Databricks. Databricks is a company that has been created by the same programmers who developed Spark and is built on top of it. What this means is that while Spark provides you with the software that is able to manage the cluster and perform the analysis, you still need the processing power, a functioning cluster, and even a platform that is able to manage everything for you. The answer is found in the form of Databricks, which offer several solutions for using Apache Spark. Some of the main Cloud platforms like Azure, for example, have implemented Databricks by integrating it in some of their tools. Azure Databricks is one of the most common tools used to deal with Spark in the Azure ecosystem.

Databricks logo

If you have any knowledge of programming, you might have already understood that Spark is a library that is much more complex than the current solutions available on the market. Even the most common ones, and open source. One example is pandas. This library is useful when you are dealing with very little dataset because you you only need one machine to run everything. However, as mentioned before, you cannot use pandas when dealing with a cluster. If you have 100GB of data, you need several computers to run your application. Because pandas is definitely not the right answer, this should give you an idea of the difference between using simple tools and more complex ones.

Conclusion

Spark remains one of the most advanced software on the market, and if you wish to improve your skills as a data scientist, you might want to start studying it at one point in your career. Some of the top companies in the world require data scientists to know how to use Spark or alternative tools for Big Data analytics. I myself can confirm that a Spark is a valuable tool in my curriculum.