All about BIG DATA and Hadoop
By: Aarfah Ahmad
Tags: Hadoop Ecosystem, Hadoop, Spark, Big Data Analytics, Big Data

Hola!

This article is about BIG DATA and Hadoop stack.

I always used to get so startled by this term “BIG DATA”, it occurred to me about something too complex to understand.

Although I believed it must be surely about the size; i.e. huge or complex data as inferred by the name :P

Apparently I was right to some extent.

Big Data can be defined as a collection of complex unstructured or semi-structured data sets which have the potential to deliver actionable insights.

The term “big data” refers to data that is so large, fast or complex that it’s difficult or impossible to process using traditional methods.

The four Vs of Big Data which identifies it are –

Volume — Talks about the amount of data

Variety — Talks about the various formats of data

Velocity — Talks about the ever increasing speed at which the data is growing

Veracity — Talks about the degree of accuracy of data available

We come across such kinds of data in our day to day life but are not able to realize the impact of it. Let’s understand through some examples.

Online ticket bookings, which includes your Rail tickets, Flight tickets, movie tickets etc.

Online shopping which is your Amazon, Flipkart, Walmart, Snap deal and many more.

Data from social media sites like Facebook, Instagram, WhatsApp and a lot more.

The employee details of any Multinational Company.

Just think about the number of posts you see as soon as you open your Instagram app. At one instance there is so much that is happening, people uploading stories, going live, posting comments, likes and so much more…ever imagined how all of this gets stored, processed and presented to you and still nothing goes out-of-whack.

All this processing of data can’t be done in an efficient manner with the help of traditional methodology such as RDBMS, we need something comprehensive and effective to control such kind of data.

Big data technologies are divided into 4 fields which are classified as follows with a few examples:

Now, talking about the big data tools which make all this happen. When we talk about Big Data components, it is about a collection of components that make the entire big data handling work. People often get confused in these terms, can choosing any one of the components accomplish their goal or do they need a combination of these.

Let’s try to understand the role of each.

In this article we will talk more about Hadoop and its ecosystem.

Hadoop

Hadoop is a framework.

In simple terms, Hadoop is an open-source software framework for storing data and running applications on clusters of commodity hardware.

Framework — we hear this term quite often, what exactly does it mean…

Framework is the basic structure intended to serve as a support or guide for the building of something.

A framework, or any software framework, is a platform for developing software applications. It provides a foundation on which software developers can build programs for a specific platform.

Hadoop as a framework is used for storing, processing, and analyzing complex unstructured data sets for deriving insights and intelligence.

It provides massive storage for any kind of data, enormous processing power and the ability to handle virtually limitless concurrent tasks or jobs. Hadoop enables processing of large data sets which reside in the form of clusters. Being a framework, Hadoop is made up of several modules that are supported by a large ecosystem of technologies.

The main backbone of Hadoop processing is MapReduce which facilitates the logic of processing. MapReduce is a software framework for easily writing applications that process the vast amount of structured and unstructured data stored in the Hadoop Distributed File system.

In layman terms, it works in a divide-and-conquer manner and runs the processes on the machines to reduce traffic on the network.

Below is a picture of the Hadoop components also mentioning the alternatives one can use.

Hadoop Stack

HDFS: It stands for Hadoop Distributed File System and it is the storage unit of Hadoop.

YARN: It stands for Yet Another Resource Negotiator. It handles the cluster resource management. Allocates RAM, memory and other resources to different applications.

Sqoop and Flume for data collection and ingestion: Sqoop is used to transfer data between Hadoop and external datastores such as relational databases and enterprise data warehouses.

Flume is a distributed service for collecting, aggregating and moving large amounts of log data.

Pig: Pig is used to analyze data in Hadoop. It provides a high level data processing language to perform numerous operations on the data.

Hive: Hive is a data warehouse infrastructure tool to process structured data in Hadoop.

Hive is an ETL and data warehouse tool on top of the Hadoop ecosystem and used for processing structured and semi structured data. Hive is a database present in the Hadoop ecosystem that performs DDL and DML operations, and it provides flexible query language such as HQL for better querying and processing of data.

Hive vs Pig

Both Hive and Pig are excellent data analysis tools — one is not necessarily better than the other, but they do have different capabilities and features. Depending on your job role, business requirements, and budget, you can choose either of these Big Data analysis platforms.

Hive is used for structured data only whereas Pig for structured, unstructured and semi structured.

Pig doesn’t support partitioning of data, Hive does

Pig is faster than Hive

Pig is used for programming and Hive for reporting and analysis.

Spark: Spark is a powerful open-source distributed computing engine for processing and analyzing huge volumes of real time data. Spark can also be used for batch processing.

Then what’s the difference between Spark and Hadoop.

Hadoop vs Spark

Spark, just like Hadoop MapReduce, also works with the system to distribute data across the cluster and process the data in parallel.

What makes Spark different and more powerful(100x faster) than Hadoop is the in-memory processing capability .

. Hadoop is designed to handle batch processing efficiently whereas Spark is designed to handle real-time data efficiently.

Hadoop is a high latency computing framework, which does not have an interactive mode whereas Spark is a low latency computing and can process data interactively.

Spark quickly rose to popularity as a replacement for the Apache Hadoop™ MapReduce paradigm in a large part because it easily connected to a number of different data sources. Most important among these data sources was the Hadoop Distributed File System (HDFS).

Mahout: Mahout is used to create scalable and distributed machine learning algorithms. It has a library that contains in-built algorithms for collaborative filtering, classification and clustering.

Ambari: Ambari is an open-source tool responsible for keeping track of running applications and their statuses.

Kafka: Kafka is a distributed streaming platform to store and process streams of records. It builds real-time streaming data pipelines that reliably get data between applications.

It is an open-source stream-processing software platform developed by the Apache Software Foundation, written in Scala and Java.

Storm: Storm is a processing engine that processes real-time streaming data at a very high speed. It has the ability to process over a million jobs in a fraction of seconds on a node.

Oozie: Oozie is a workflow scheduler system to manage Hadoop jobs. It has 2 parts: Work engine and Coordinator engine.

Stages of Big Data Processing

With so many components within the Hadoop ecosystem, it can become pretty intimidating and difficult to understand what each component is doing. Therefore, it is easier to group some of the components together based on where they lie in the stage of Big Data processing.

There are four stages of Big Data processing: Ingest, Processing, Analyze, Access. Let us look at them in detail.

Ingest

The first stage of Big Data processing is Ingest. The data is ingested or transferred to Hadoop from various sources such as relational databases, systems, or local files. Sqoop transfers data from RDBMS to HDFS, whereas Flume transfers event data.

Flume, Kafka, and Sqoop are used to ingest data from external sources into HDFS

2. Processing

The second stage is Processing. In this stage, the data is stored and processed. The data is stored in the distributed file system, HDFS, and the NoSQL distributed data, HBase.

Spark and MapReduce perform the data processing. MapReduce and Spark are used to process the data on HDFS and perform various tasks

3. Analyze

The third stage is Analyze. Here, the data is analyzed by processing frameworks such as Pig, Hive, Spark and Impala.

Pig converts the data using a map and reduce model and then analyzes it. Hive is also based on the map and reduce programming and is most suitable for structured data.

4. Access

The fourth stage is Access, which is performed by tools such as Hue and Cloudera Search. In this stage, the analyzed data can be accessed by users. Hue is the web interface whereas Cloudera Search provides a text interface for exploring data.

Oozie helps to schedule tasks. Since it works with various platforms, it is used throughout the stages.

Zookeeper synchronizes the cluster nodes and is used throughout the stages as well.

You can also refer this to learn the Commonly Used Services in AWS to used to build out your big data framework in AWS.

Hope this was helpful!

Thanks for reading! Please do share the article, if you liked it. Any comments or suggestions are welcome!