Big data and Hadoop
By: M S Dilli
Tags: Hadoop Training, Big Data, Hadoop, Big Data Analytics

Big Data is generally considered to be very huge amount of data for storing and processing. Data in huge volume and different varieties can be considered as Big Data. Data is changing our world and the way we live at an unprecedented rate. Big data is the new science of analyzing and predicting human and machine behavior by processing very huge amount of related data. Big data refers to speedy growth in the volume of structured, semi-structured and unstructured data. It is estimated to generate 50,000 GB data per second in year 2019. Today’s enterprises are generating massive amount of Data, Which essentially has 3 attributes :

Volume : — The size of the data, we are talking about GB and TBs here

Velocity : — The rate at which the data is being generated

Variety :- Data from Multiple sources and multiple kinds

This added complexity can not be handled with traditional frameworks hence Hadoop ( not the only solution ) Hadoop is a parallel processing programming framework which work in MapReduce. Apache Hadoop is one of the tool to work on Big Data. It is open source, software framework that runs on a commodity hardware.

Why we are talking about Hadoop?

We are talking because, On a hadoop framework we can store any type of data weather it is a Structured, Semi Structured and Unstructured in Hadoop Distributed File System [HDFS] layer.

Apache hadoop is open source, it means you don’t need to pay for license while using commercially.

Apache Hadoop runs on commodity hardware, it means you don’t need to stick to a single vendor. You can choose any vendor, who offers infrastructure at low cost.

Consider a problem where you need to count the number of words in a 5 pound book. would be very difficult for one person but if you tear the pages and distribute it to hundreds of people. Each will count the words in their “page” and then you can simply total that count from each person you shall have to total word count in no time.

Why Hadoop with an real time example ?

Suppose you have a very big file (50 gb log file for an example) and you want to parse it , do some filtering on it and see the result. What are the options?

1. If you have a computer with ram more that 64 gb ( assuming the additional 16 gb will be used for os and other processes) you can write some code to get it done. Still it will be super slow. And if the file is even larger (in petabyte scale) it wont be even feasible. Rams on petabye scale is not available yet

2. Parse the file in smaller files (may be 10000 files , each in the scale of megabytes) and read them sequentially

3. Using approach 2 but using multiple threads , each thread reading a smaller file and finally merge the threads and compute the result

Hadoop is just step 3 only with a distributed computing touch to it. You have bunch of computers. One of the computer is master node and the rest are slave nodes.All these nodes form a cluster. This is hdfs or hadoop distributed file system. You upload a huge file to the cluster. This huge file gets divided in small file chunks of some size ( each X megabyte for an example). These chunks are replicated throughout the cluster following a replication factor. Then using a programming framework called mapreduce , you do operations on the content of the file chunks and get the desired result.

Hadoop Ecosystem:

It comprises of various tools that are required to perform different tasks in Hadoop.

These tools provide you a number of Hadoop services which can help you handle big data more efficiently.

There are the popular tools that are part of the Hadoop Ecosystem:

HDFS: It stands for Hadoop Distributed File System and it is the storage unit of Hadoop. Have a detailed explanation on HDFS and how the data is read and write into hdfs.

YARN: It stand for Yet Another Resource Negotiator. It handles the cluster resource management. Allocates RAM, memory and other resources to different applications.

MapReduce: MapReduce processes large volumes of data in a parallelly distributed manner.

HBase: It is a column-oriented non-relational database management system that runs on top of Hadoop Distributed File System (HDFS). HBase provides a fault-tolerant way of storing sparse data sets, which are common in many big data use cases. It is well suited for real-time data processing or random read/write access to large volumes of data.

Sqoop and Flume for data collection and ingestion: Sqoop is used to transfer data between Hadoop and external datastores such as relational databases and enterprise data warehouses.

Flume is distributed service for collecting, aggregating and moving large amounts of log data.

Pig: Pig is used to analyze data in Hadoop. It provides a high level data processing language to perform numerous operations on the data.

Hive: Hive facilitates reading, writing and managing large datasets residing in the distributed storage using SQL (Hive Query Language).

Spark: Spark is an open-source distributed computing engine for processing and analyzing huge volumes of real time data.

Mahout: Mahout is used to create scalable and distributed machine learning algorithms. It has a library that contains in-built algorithms for collaborative filtering, classification and clustering.

Ambari: Ambari is an open-source tool responsible for keeping track of running applications and their statuses.

Kafka: Kafka is a distributed streaming platform to store and process streams of records. It builds real-time streaming data pipelines that reliably get data between applications.

Storm: Storm is a processing engine that processes real-time streaming data at a very high speed. It has the ability to process over a million jobs in a fraction of seconds on a node.

Oozie: Oozie is a workflow scheduler system to manage Hadoop jobs. It has 2 parts: Work engine and Coordinator engine.