What is big data and big data analytics and spark project set up with Scala
By: Ganesh Singh
Tags: Big Data Analytics, Big Data, Spark, Configuration, Setup

What is big data and big data analytics and spark project set up with Scala Ganesh Singh Apr 23, 2020·3 min read

What is big data

I’m going to give you simplest definition of big data. Data

Which is rapidly increasing with time, volume and velocity

Which cannot be stored in one device or system it required cluster of devices

Which cannot be processed with on system it also required cluster of system

Now a days every second generate many petabytes of data so it required data management and analytics tools to manage that data and it’s valuable information

Google, Facebook and Amazon many more are investing billions of dollar to analyse that data and find the valuable information

Types of Big Data:

Big Data can be broadly classified into three categories:

Structured

Any data which you can store, process and access in a fixed format can be classified as structured data. The data is already stored in databases in an ordered manner. The format of data and how to derive value out of it is priory known.

Example of structured data: Information stored in any database software

2. Unstructured

Any data which has an unknown structure or format can be termed as unstructured data. The data size is massive and it is not easy to derive value out of it. The data can contain a mix of text files, videos and images.

Example of unstructured data: The output for any Google search

3. Semi-structured

It contains both the above forms of data. Many a time, if the data is defined but not structured, it can be classified as semi-structured data. Semi-structured data contains information which contains organizational properties but is not in the traditional database format.

Example of semi-structured data: Any data stored in an XML file

Three V’s Of Big Data

In the early 2000 ’s, Gartner analyst, Doug Laney articulated the concept of Big Data in the form of three V’s. Also, as Big Data comprises of data creation, storage and retrieval; it can be characterised remarkably in terms of:

1. Volume

The definition of Big Data itself denotes a copious amount of data. All businesses have huge amounts of data which includes data collected from — business deals, transactions and investments, social media stats and other data as well. Thus, the volume of data is crucial as it gives an idea about how to extract value out of the data.

Hence, volume is one important parameter which has to be considered when working with Big Data.

2. Velocity

Velocity refers to the speed at which data is generated. Generating includes both; receiving of data and the action performed on the data. How fast the data is generated and processed determines the actual potential of the data.

Also, the stream of generated data is unprecedented and massive.

Processing and functioning with this unprecedented data stream will determine the real potential of the data. That is why velocity is crucial to Big Data.

3. Variety

Nowadays, when it comes to data, we are not just limited to plain text data or structured data in the form of databases. Data means different types of data — structured, unstructured, numeric, audio, video, pdf, financial transactions and ticker data. All these different types of data require different processing and correct handling to derive context out of it.

This is where big data tools comes in picture. Based on what you want to do with that data you can decide you tool like Hadoop, spark, pig , Cassandra, HBase and zookeeper etc.

Each tool has its pros and cons you can decide based on your requirements.

What can Big Data analytics do? (What is Big Data Analytics and Why is it Important? ) or (How it can be used and why? )