Big Data: The movie
By: Roop Sannigrahi
Tags: Hdfs Architecture, Hadoop, Hdfs, Mapreduce, Big Data

Let the count down begin

Prologue:

It all started way back, when world wide web (WWW) was not made. There was a variety of data even that time, but it was all physical data and very little of it was digital. Then came the introduction of WWW which revolutionized the world. Data stared flooding in and it just kept growing and it was difficult to manage it effectively and efficiently. The term that was coined for this type of data was “Big Data” by Roger Mougalas in 2005.

Present day: How the Big Data is managed?

Now people don’t bother much about how much data is being transferred , managed, displayed on the web. There have been many tools that has made this possible since the term “Big data” was coined. One of the biggest and widely used tool is Hadoop.

Hadoop is a java based framework, it is an open-source framework. Hadoop is used for storing and processing big data. In Hadoop data is stored on inexpensive commodity servers that run as clusters. It is a distributed file system allows concurrent processing and fault tolerance. Hadoop MapReduce programming model is used for faster storage and retrieval of data from its nodes.

Hadoop is developed by Doug Cutting and Michale J in 2006. It is managed by apache software foundation and licensed under the Apache license 2.0 Hadoop is very useful for the big business because it is based on cheap servers so it required less cost to store the data and processing the data. Hadoop helps to make a better business decision by providing a history of data and various record of the company, So by using this technology company can improve its business. Hadoop does lots of processing over collected data from the company to deduce the result which can help to make a future decision.

Today’s world is heavily depended on data and most of the technological companies work with data all over the world. Hadoop is a tool used for this purpose and it has many features and modules. The following are its modules that makes up Hadoop:

Hadoop common- it contains libraries and modules required by other Hadoop modules.

Hadoop Distributed file systems(HDFS)- It is a distributed file-system that stores data on commodity machines, providing very high aggregate bandwidth across the cluster.

Hadoop YARN- It’s platform responsible for managing computing resources in clusters and using them for scheduling users’ applications.

Hadoop MapReduce- It’s an implementation of the MapReduce programming model for large-scale data processing.

Hadoop Ozone- An object store for Hadoop.

There have been various implementation of this technology. It was bought by Apache and is currently known as Apache Hadoop. All of its management is taken over managed by Apache which is an amazing web company and is primarily Known for things like LAMP (Linux Apache MySQL and PHP), Hadoop etc.

The technical story: HDFS, MapReduce and architecture

Just bear with me

Hadoop Distributed file System (HDFS):

Hadoop Distributed File System (HDFS) is the storage component of Hadoop. Hadoop File System was developed using distributed file system design. It is run on commodity hardware. Unlike other distributed systems, HDFS is highly fault tolerant and designed using low-cost hardware. HDFS holds very large amount of data and provides easier access. To store such huge data, the files are stored across multiple machines. These files are stored in redundant fashion to rescue the system from possible data losses in case of failure. HDFS also makes applications available to parallel processing.

Some Important Features of HDFS(Hadoop Distributed File System):

It’s easy to access the files stored in HDFS.

HDFS also provide high availability and fault tolerance.

Provides scalability to scaleup or scaled own nodes as per our requirement.

Data is stored in distributed manner i.e. various Data nodes are responsible for storing the data.

HDFS provides Replication because of which no fear of Data Loss.

HDFS Provides High Reliability as it can store data in the large range of Petabytes.

HDFS has in-built servers in Name node and Data Node that helps them to easily retrieve the cluster information.

Provides high throughput.

HDFS stores the data in the form of the block where the size of each data block is 128MB in size which is configurable means you can change it according to your requirement

Hadoop MapReduce:

MapReduce is a processing technique and a program model for distributed computing based on java. The MapReduce algorithm contains two important tasks, namely Map and Reduce. Map takes a set of data and converts it into another set of data, where individual elements are broken down into tuples. Secondly, reduce task, which takes the output from a map as an input and combines those data tuples into a smaller set of tuples.

Algorithm for MapReduce:

Data is sent to a computer where data is actually stored. There are three stages to MapReduce: Map stage, shuffle stage and reduce stage

Map Stage : The map or mapper’s job is to process the input data. Generally the input data is in the form of file or directory and is stored in the Hadoop file system (HDFS). The input file is passed to the mapper function line by line. The mapper processes the data and creates several small chunks of data.

: The map or mapper’s job is to process the input data. Generally the input data is in the form of file or directory and is stored in the Hadoop file system (HDFS). The input file is passed to the mapper function line by line. The mapper processes the data and creates several small chunks of data. Reduce stage: This stage is the combination of the Shuffle stage and the Reduce stage. The Reducer’s job is to process the data that comes from the mapper. After processing, it produces a new set of output, which will be stored in the HDFS.

During this process Hadoop sends the Map and reduce job to the appropriate servers. It is the frame work that manages all the details of data-passing such as issuing tasks, verifying task completion, and copying data around the cluster between the nodes. To reduce the data traffic most of the data tasks are done locally. After the task is done the cluster gathers the reduced data and sends is back to the servers.

Architecture:

Hadoop has a Master-Slave Architecture for data storage and distributed data processing using MapReduce and HDFS methods.

Name node: Name node represented every files and directory which is used in the namespace

Data node: Data node helps you to manage the state of an HDFS node and allows you to interacts with the blocks

Slave node: The slave nodes are the additional machines in the Hadoop cluster which allows you to store data to conduct complex calculations. Moreover, all the slave node comes with Task Tracker and a Data node. This allows you to synchronize the processes with the Name node and Job Tracker respectively.

Master node: The master node allows you to conduct parallel processing of data using Hadoop MapReduce.

Climax : Applications of this technology

Here we go

Customer’s requirement understanding: Hadoop’s most important uses are in Customer’s requirement understanding. Many companies like financial, telecom use this technology to find out the customer’s requirement by analysis big amount of data and extracting this important information from these data, social media also uses this technology, it keeps on posting an advertisement on various social media sites to the target customer whenever the user open their social media on browser, Credit card company uses this technology to find out the exact customer for their product, they contact the customer through various way to sell their product ,Many online booking sites keeps the track of users for their previous journey according to this information or based on customer interest they suggest to their customer flights on these basis , Many e-commerce company keeps track of the product which was bought by together, on basis of this they suggest to customer to buy other product when customer trying to buy one of relevant product from that group, for example, mobile back cover, screen guard suggestion on buying of the mobile. Security and law enforcement: It is used to detect and prevent cyber-attacks. Police forces use big data tools to catch criminals and even predict criminal activity and credit card companies use big data use it to detect fraudulent transactions. Hadoop has a big amount of data that is used to extract meaningful information from this data. A financial company uses these techniques to search the customers. Finance: Financial organizations use Hadoop for fraud detection and prevention. They use Apache Hadoop for reducing risk, identifying rogue traders, analyzing fraud patterns. Hadoop helps them to precisely target their marketing campaigns on the basis of customer segmentation. Hadoop helps financial agencies to improve customer satisfaction. Credit card companies also use Apache Hadoop for finding out the exact customer for their product. Understanding and optimizing business processes: Another most common reason, why the uses of Hadoop are important is because it is also used in business process.it has optimized the performance of the company in a various way. Retailer can customize their stocks by prediction came from various source like social media google search, various other platforms, on basis of this company can take best decision to improve their business and maximized their profits, radio frequency and geographical positioning system can help to track the product and take proper action to minimized the loss of the company ,it helps to give best-optimized rout for the product transfer from one place to another country and between interstate this helps the company to cost-cutting and improving their performance, Many companies uses this technology to improve their workspace by monitoring the behavior of their employees. It put the sensor into badges of an employee to monitor their interaction with other people of the organization, this helps lots to take HR decision in case of any issue between the employees. Personal quantification: Hadoop is used to improve personal life, it provides many ways to improve day to day life by monitoring the sleep pattern, morning walk, a whole daily routine of healthy people. By taking of conclusion from all these patterns it helps to improve our lives. Many dating sites currently using Hadoop to find out the people with common interest it helps to find out true love, this is one of a big field in applications of Hadoop. It helps to improve our health by giving guidelines and suggestion.

End credits:

Well, this was the story of how Big data is managed using one of the most powerful tools and its various application. we saw the technicalities of Hadoop and along the way we got to know some history.

Thank you for watching(reading) this movie.

sources .. uhh literally many google searches and YouTube, to much to list ;-)

Please react and comment to let me know your thoughts, and what other topics or trends you would like to read.