Big Data Analytics: Apache Spark vs. Apache Hadoop
By: Nhi Diep
Tags: Big Data, Big Data Analytics, Data Science, Apache Spark

Big Data Analytics: Apache Spark vs. Apache Hadoop

What is Big Data?

“Big data is high-volume, high-velocity and high-variety information assets that demand cost-effective, innovative forms of information processing for enhanced insight and decision making”[11]

Big data analytics can be time-consuming, complicated, and computationally demanding, without the proper tools, frameworks, and techniques. When the volume of data is too high to process and analyze on a single machine, Apache Spark and Apache Hadoop can simplify the task through parallel processing and distributed processing. To understand the need for parallel processing and distributed processing in big data analytics, it is important to first understand what “big data” is. The high-velocity at which big data is generated requires that the data also be processed very quickly and the variety of big data means it contains various types of data, including structured, semi-structured, and unstructured data [4]. The volume, velocity, and variety of big data calls for new, innovative techniques and frameworks for collecting, storing, and processing the data, which is why Apache Hadoop and Apache Spark were created.

Parallel Processing vs. Distributed Processing

Understanding what parallel processing and distributed processing is will help to understand how Apache Hadoop and Apache Spark are used in big data analytics. Since both parallel processing and distributed processing both involve breaking up computing into smaller parts, there can be confusion between the two. The difference between parallel computing and distributed computing is in the memory architecture [10].

“Parallel computing is the simultaneous use of more than one processor to solve a problem” [10]. “Distributed computing is the simultaneous use of more than one computer to solve a problem” [10].

Parallel computing tasks access the same memory space, while distributed computing tasks don’t since distributed computing is disk-based, instead of memory-based. Some distributed computing tasks run on one computer, and some on others [9].

Apache Spark vs. Apache Hadoop

Apache Hadoop and Apache Spark are both open-source frameworks for big data processing with some key differences. Hadoop uses the MapReduce to process data, while Spark uses resilient distributed datasets (RDDs). Hadoop has a distributed file system (HDFS), meaning that data files can be stored across multiple machines. The file system is scalable, since servers and machines can be added to accommodate increasing volumes of data [2]. Spark does not provide a distributed file storage system, so it is mainly used for computation, on top of Hadoop. Spark does not need Hadoop to run, but can be used with Hadoop since it can create distributed datasets from files stored in the HDFS [1].

Performance Differences

A key difference between Hadoop and Spark is performance. Researchers from UC Berkeley realized Hadoop is great for batch processing, but inefficient for iterative processing, so they created Spark to fix this [1]. Spark programs iteratively run about 100 times faster than Hadoop in-memory, and 10 times faster on disk [3]. Spark’s in-memory processing is responsible for Spark’s speed. Hadoop MapReduce, instead, writes data to a disk that is read on the next iteration. Since data is reloaded from the disk after every iteration, it is significantly slower than Spark [7].

Apache Spark RDDs

Apache Spark’s fundamental data structure, the resilient distributed dataset (RDD), is a fault-tolerant, immutable, and distributed collection of objects that can be processed in parallel across a cluster. RDDs are resilient due to their fault tolerance, which means that it has recovery from node failure. In the event that a node fails to complete its task, the RDD can be automatically reconstructed on the remaining nodes to complete the task [6]. RDDs’ immutability helps to prevent and avoid data inconsistency, even when performing transformations on unstructured data [9]. Big data is often variable and can include text data, relational databases, videos, audio, photos, and graph data [2]. RDD’s make it possible to process both structured and unstructured data efficiently. DataFrames, which were built on top of RDDs, are also immutable, distributed collections of data, but organized into columns and rows, similar to a relational database[5]. Spark DataFrames make it even easier to process and analyze data from a variety of sources and formats, such as JSON files and Hive tables [1].

Spark Streaming

Apache Spark’s stack includes multiple frameworks for big data processing, such as Spark Streaming. The Spark Streaming framework is for stream processing fault-tolerant, live data streams to handle big data’s velocity [8]. Upon receiving live input data, Spark Streaming divides the data into batches, so that the data is processed in batches, and the final result is a stream of processed batches [5]. Spark Streaming data is known as Discretized Streams (DStreams), which essentially are sequences of RDDs [6]. DStreams are consistent and fault tolerant. Big data is valuable to businesses and decision-makers due to its ability to be processed in real-time or near real-time. An example use-case of Spark Streaming is an investment firm needing to monitor social media trends that could impact their investments and holdings. Staying up to date on what is being said about the firm’s clients or companies they have holdings in can greatly impact business outcomes.

Spark MLlib

Apache Spark has a machine learning library called MLlib that provides the major machine learning algorithms such as classification, clustering, regression, dimensionality reduction, transformers, and collaborative filtering [7]. Some machine learning algorithms can be applied to streaming data, which is useful for applications that use Spark Streaming [6]. An example of a classification algorithm being used on a Spark Streaming application is credit card fraud detection. As soon as a new incoming transaction is processed, the classification algorithm will classify whether or not a transaction is fraud. Identifying potentially fraudulent transactions as soon as they happen is important so that banks can reach out to the customer as soon as possible. The challenging endeavor involved with Spark Streaming is the unpredictability of the volume of the incoming data [8]. The inconsistent distribution of incoming data over the stream’s lifetime can make stream learning for machine learning models rather difficult.

Recommendation Systems

A popular real-world use for Spark Streaming and MLlib is for building machine learning-based recommendation systems. Recommendations systems have impacted how we shop online, the movies we watch, the songs we listen to, and the news we read. Companies most notable for their recommendation systems include Netflix, YouTube, Amazon, Spotify, and Google. With companies as large as these, generating and collecting as much data as they do, big data analytics is amongst the top of their priorities. MLlib includes an iterative collaborative filtering algorithm, known as Alternating Least Squares to construct recommendation systems [6]. Collaborative filtering systems make recommendations based on similarity measures between users and items [8]. Similar items will be recommended to similar users. The system doesn’t know details about the items, only that similar users viewed or chose the same item.

Apache Mahout

MapReduce once had its own machine learning library, however, since MapReduce is inefficient for iterative processing, it quickly lost its compatibility with the library to Apache Spark. Apache Mahout is the machine learning library built on top of Apache Hadoop that started out as a MapReduce package for running machine learning algorithms. Since machine learning algorithms are iterative, MapReduce encountered scalability and iterative processing issues [6]. Because of these issues, Apache Mahout stopped supporting MapReduce-based algorithms, and started supporting other platforms, such as Apache Spark.

Conclusion

From its inception, Apache Spark was developed to be fast, and address Apache Hadoop’s shortcomings. Apache Spark is not only faster, but also uses in-memory processing and has many libraries built on top of it to accommodate for big data analytics and machine learning. Despite Hadoop’s shortcomings, both Spark and Hadoop play major roles in big data analytics and are harnessed by big tech companies around the world to tailor user experiences to customers or clients.

Works Cited

[1] Ankam, Venkat. Big Data Analytics. Packt Publishing, 2016. EBSCOhost,search.ebscohost.com/login.aspxdirect=true&db=nlebk&AN=1364660&site=eho st-live.

[2] “Apache Hadoop.” ​Apache Hadoop,​ hadoop.apache.org/.

[3] “Apache Spark.” ​Apache Spark, ​https://spark.apache.org/.​

[4] Balazinska, Magda and Dan Suciu. ​Parallel Databases and MapReduce. ​Spring 2016, http://pages.cs.wisc.edu/~paris/cs838-s16/lecture-notes/lecture-parallel.pdf​. PowerPoint Presentation.

[5] Drabas, Tomasz, et al. ​Learning PySpark​. Packt Publishing, 2017. ​EBSCOhost​,search.ebscohost.com/login.aspx?direct=true&db=nlebk&AN=1477650&site=ehost-live.

[6] Dua, Rajdeep, et al. ​Machine Learning with Spark — Second Edition.​ Vol. Second edition, Packt Publishing, 2017. ​EBSCOhost​, search.ebscohost.com/login.aspxdirect=true&db=nlebk&AN=1513368&site=eho st-live.

[7] Hari, Sindhuja. “Hadoop vs Spark: Head to Head Comparison Choosing Right Framework.” ​Hackr.io,​ 17 Dec. 2019, ​https://hackr.io/blog/hadoop-vs-spark​.

[8] Ignacio, Franco Galeano Manuel. ​Big Data Processing with Apache Spark: Efficiently Tackle Large Datasets and Big Data Analysis with Spark and Python.​ Packt, 2018.

[9] Pusukuri, Kishore. ​CS5412 / Lecture 25 Apache Spark and RDDs.​ Spring 2019, http://www.cs.cornell.edu/courses/cs5412/2019sp/slides/Lecture-25.pdf​. PowerPoint Presentation.

[10] Pierfederici, Francesco. ​Distributed Computing with Python.​ Packt Publishing, 2016. ​EBSCOhost,​search.ebscohost.com/login.aspx?direct=true&db=nlebk&AN=1220461&site=ehost-live.

[11] “What Is Big Data.” ​Big Data Alliance​, https://www.bigdata-alliance.org/what-is-big-data/.