BIG DATA
By: Sree Sowmya
Tags: Hadoop, Big Data

What is big data

• Big data is also data with a huge size

• Big data is a term used to describe collection of data that is huge in volume and yet growing exponentially with time

• In short such data is so large and complex that none of the traditional data management tools are able to process it

• Machine learning is the largest source of big data

• Three types of diverse data sources are machine data, Organizational data and people

• An example of machine data is Weather station sensor output

• An example of Organizational data is Disease data from Center for Disease Control

Advantages of big data

• Real-time Notification enables Real-time Actions

• Design Differently

• Culture shift to real time operations

• Increased use of scalability computing

• Supervisory control and Data Acquisition(SCADA)

Big data generated by people : Unstructured data

• People are generating data at a faster pace from Facebook, Twitter, Blogger, YouTube, Documents and Internet searches

• This data is text heavy and unstructured

• There is a huge growth in volume of data

• 80%-90% entire data is unstructured

• The costs and time of process of acquiring, storing, cleaning, retrieving and processing unstructured data can add up to more investment before we start reaping value from this process

• It is hard to find tools and people to implement such a process and reap value at the end

Big data generated by people: How is it being used

• Big data tools are designed from scratch to manage unstructured information and analyze it

• Hadoop is designed to support the processing of large datasets in a distributed computing environment. Hadoop can handle big batches of distributed information but there is a need for real time processing of people generated data like Twitter or Facebook updates

• Social media and market data are two types of high velocity data

• Storm and Spark are two open source frameworks that handle such real time data generated at a fast rate. Both Storm and Spark can integrate data with any database or data storage technology

• Data warehouses are central repositories of integrated data from one or more sources. The data that gets stored in warehouses, gets extracted from multiple sources

Organization generated data: Structured

• Big data generated by organizations is structed but siloed

• Each organization has distinct operation practices and business models, which result in a variety of data generation platforms

• Some common types of organizational big data come from commercial transactions, credit cards, government institutions, e-commerce, banking or stock records, medical records, sensors, transactions , clicks.

• Almost every event can be potentially stored. Organizations store this data for current and future use as well as for analysis of the past.

• Highly structured organizational data is very useful and trustworthy, and thus a valuable source of information, organizations must pay special attention to breaking up the silos of information to make use of its potential.

Organization generated data-Benefits come from combining with other types

Future Trends for Organizations

• Efficient Operations

• Higher sales

• Improved Safety

• Customer Satisfaction

• Better profit margins

• Improved Product placement

Characteristics of Big Data

• Volume

• Variety

• Velocity

• Veracity

• Valence

• Value

Hadoop

• The Hadoop ecosystem is great for Big data

• Enable Scalability

• Handle Fault tolerance

• Optimized for a Variety of Data Types

• Facilitate a shared environment

• Provide Value

What is Hadoop

• Apache Hadoop is an open source software framework for storage and large scale processing of data-sets on clusters of commodity hardware

• Hadoop was created by Doug Cutting and Mike Cafarella in 2005 Named the project after son’s toy elephant

• Moving Computation to Data

• Scalability at Hadoop’s core!

• Reliability!

• New Approach to Data

• New Kinds of Analysis

Apache Framework Basic Modules

• Hadoop Common

• Hadoop Distributed File System (HDFS)

• Hadoop YARN

• Hadoop MapReduce

The Basic Hadoop Components

• Hadoop Common — libraries and utilities

• Hadoop Distributed File System (HDFS) — a distributed file-system

• Hadoop YARN — a resource-management platform, scheduling

• Hadoop MapReduce — a programming model for large scale data processing

Hadoop Applications and Frameworks

• HBase — a scalable data warehouse with support for large tables

• Hive — a data warehouse infrastructure that provides data summarization and ad hoc querying

• Pig — A high-level data-flow language and execution framework for parallel computation

• Spark — a fast and general compute engine for Hadoop data. Wide range of applications — ETL, Machine Learning, stream processing, and graph analytics.

Original HDFS Design Goals

• Resilience to hardware failure

• Streaming data access

• Support for large dataset, scalability to hundreds/thousands of nodes with high aggregate bandwidth

• Application locality to data

• Portability across heterogeneous hardware and software platforms

Original HDFS Design

• Single NameNode — a master server that manages the file system namespace and regulates access to files by clients.

• Functions

• Serving read/write requests from clients

• Multiple DataNodes — typically one per node in the cluster.

• Manage storage

• Block creation, deletion, replication based on instructions from NameNode

HDFS in Hadoop 2

• HDFS Federation

• Multiple namespaces

• Heterogeneous Storage and Archival Storage : ARCHIVE, DISK, SSD, RAM_DISK

• Multiple Namenode servers

• High Availability — redundant NameNodes

MapReduce Framework

• Software framework — for writing parallel data processing applications

• MapReduce job splits data into chunks

• Map tasks process data chunks

• Framework sorts map output

• Reduce tasks use sorted map data as input

Original MapReduce Framework

• Single master JobTracker

• JobTracker schedules, monitors, and re-executes failed tasks.

• One slave TaskTracker per cluster node

• TaskTracker executes tasks per JobTracker requests.

YARN: NexGen MapReduce

• Main idea — Separate resource management and job scheduling/monitoring.

• Global ResourceManager (RM)

• NodeManager on each node

• ApplicationMaster — one for each application

YARN FEATURES

• High Availability ResourceManager

• Timeline Server

• Use of Cgroups

• YARN — web services REST APIs

• Secure Containers

Hadoop Execution Environment

• Execution frameworks: YARN, Tez, and Spark

Apache HBase

• Scalable data store

• Run interactively using HBase shell.

• Non-relational distributed database

• Runs on top of HDFS

• Compression

• In-memory operations: MemStore, BlockCache

HBase Features

• Consistency

• High Availability

• Automatic Sharding

• Replication

• Security

• SQL like access (Hive, Spark, Impala)

Applications using HDFS

• Can use APIs to interact with HDFS

• Core component of Hadoop stack — used by all applications

• HBase is a good example of an application that runs on top of HDFS with good integration

• Spark can run directly on HDFS without other Hadoop components