fabiochiu/medium-articles huggingface dataset:
Proxying requests to google cloud storage using Hadoop GCS connector through Envoy proxy.
By: Mukund Thakur
Tags: Envoy Proxy, Google Cloud Storage, Hadoop, Envoy

Introduction:

Hadoop GCS connector is widely used in the community to connect from on premise Hadoop cluster and related services to google cloud storage https://github.com/GoogleCloudDataproc/hadoop-connectors

Recently I have been exploring how to use Envoy proxy https://www.envoyproxy.io/docs/envoy/latest/ as a Front proxy such that all requests flow through Envoy and we can do monitoring of requests using an external http service as well before forwarding the requests to google storage servers. So in this post I am going to talk about how to do this envoy proxy setup for this use case end to end. I will leave the monitoring using external http service flow for my next post :).

Prerequisites:

Before we start, I am assuming that we have a Hadoop client/cluster from where we are able to access our buckets present in google cloud storage. Essentially we are able to run commands like below successfully:

hadoop fs -ls gs://your_bucket_name/

Or if we are not using Hadoop connector, we should be able to access GCS buckets using curl requests for example :

Also we know to run the envoy server locally using a sample config file. For more info follow envoy docs.

Steps :

Run the envoy proxy with the below config file using command :

envoy -c config.yaml -l debug

We can change the logging level as well using -l debug/trace for logging the requests and responses on the envoy server.

Some explanation points for the above configuration:

Envoy config consists of list of filters and clusters. Filters are from accepting the requests from downstream client/server (here Hadoop client) and clusters are for forwarding the requests to upstream server( here Google storage servers).

This will start a http proxy sever at address localhost:1000.

All the requests directed towards localhost:10000 will be rewritten based on the host_rewrite_literal config and then forwarded to the specific cluster based on routes match rules.

config and then forwarded to the specific cluster based on match rules. Once the requests are rewritten, it will be sent to google storage servers.

To configure the proxy on the connector side we have to set the parameter fs.gs.storage.root.url and then run the command below. This will enable all the storage requests to flow through the proxy but the auth requests will still go directly to the google OAuth servers.

hadoop fs -Dfs.gs.storage.root.url=http://localhost:10000 -ls gs://your_bucket/

Here is the envoy log snippet:

As we can see from the logs, the flow is

The storage calls first came to envoy

`‘:authority’, ‘localhost:10000’

’path’, ‘/storage/v1/b/my_bucket’

And then got rewritten and forwarded to google storage servers.

cluster 'service_google' match for URL '/storage/v1/b/my_bucket'

':authority', 'storage.googleapis.com'

':path', '/storage/v1/b/my_bucket'

The same flow can be achieved via making the curl requests using postman to localhost:10000/storage/v1/b/gcp-public-data-landsat
Hadoop Nedir? Bileşenlerini Tanımaya Başlayalım 1 (HDFS,MapReduce)
By: Birol Emekli̇
Tags: Mapreduce, Hadoop, Hdfs, Big Data

By signing up, you will create a Medium account if you don’t already have one. Review our Privacy Policy for more information about our privacy practices.
Busting Hadoop Myth!!!
Tags: Hadoop, AWS, Myths, Big Data, Linux

Busting Hadoop Myth!!!

Does Hadoop use the parallelism concept or serialism concept to upload the split data in the data nodes ??? MishanRG Dec 21, 2020·4 min read

Hello Everyone, I hope you all are doing fine.

In this blog, we will learn how data gets uploaded in the data node in the Hadoop cluster? We can see many posts, blogs, and articles that state that Hadoop uses parallelism, and some state Hadoop uses serialism, so let's find out how Hadoop uploads data in data nodes with proof. So let’s get started…

Prerequisites

NOTE: Please refer to above link to get more clear understanding in Hadoop Cluster and its formation.

First, we created a Hadoop Cluster with 1 Name Node, 4 Data nodes, and 1 Client.

Web-UI of the Cluster

Data Node in Cluster

We then uploaded a file with the name “testhadoop.txt” from the Client to see how the data gets stored in all the data-node.

We can use the command:

# hadoop fs -put <filename>

Client Uploading Data

When we upload data from the client, we know that the Client contact the Name Node(NN), and NN provides the details of the Data Nodes, and the Client takes the data to the Data Nodes and creates replication. So we used the `tcpdump` command on all 4 data nodes to capture the package(data) coming from the client to check if all data nodes get the package (PARALLELISM) at the same time. OR the data is given to one data node, and that the data node transfers the file to another data node (SERIALISM). We also know that the data is replicated, so let's see if the data is sent to all nodes at once or one data node takes the client's data and passes it on.

We stored the output of `tcpdump` in a file to check the file with the grep command to see from which IP it received the packet. We evaluated port 50010 because the client uses that port to transfer packets.

# tcpdump -i eth0 port 50010 > <filename>

Data Node 1

Data Node 2

Data Node 3

Data Node 4

We ran this command after the client uploaded the data to the cluster. And now we check the output of the `tcpdump` command in all data nodes.

After uploading the file below image, we can see in the Web-UI that the data has been replicated in 3 different IPs among 4 IPs, which means one of the IPs didn’t get the data replication factor in our cluster is 3.

Now to check the packet received, we tried the command “cat” and tried “grep” to Client IP to see if one data node receives the packet or all of them.

CLIENT IP: 35–154–195–85
CloudHadoop 에서 Spark 3 버전 사용하기
By: Ha Ryoung Park
Tags: Cloud Hadoop, Hadoop, Spark 3, Apache Spark, Database

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
How To Configure Big Data Hadoop In Just A Few Steps Using Ansible Automated Script
By: Gupta Aditya
Tags: Ansible, Hadoop, Automation, DevOps, Ansible Playbook

Hey guys hope you all are doing good today in this article we are going to set the Hadoop name node and data node with the power of ansible.

If you have not checked my previous article on ansible I recommend you to read that first in that we have talked about how to configure yum that is necessary for any practical and in that we have also covered how to automate docker using ansible.

First, we have to set our inventory file for that we have made two groups controller and data so that when we run playbook we can specify the groups because we don’t want our data node server to set up as a name node. Once the inventory is setup repeat the same steps to set up an ansible configuration file and checking connectivity as shown in the above article.

Inventory

Once All the steps are done make a new folder for setting name node from the command:-

mkdir/folername

Core-site.xml

For setting up the name node we have to edit two files hdfs-site.xml and core-site.xml so we have to make a clone of those files in our ansible controller node.

we have use the jinja template inside the core-site.xml file.

we also need Hadoop and JDK installation file so that we can copy in target node and install them.

hdfs-site.xml

Deploying playbook

Once we are done with all required files we can deploy our playbook.

As you can see below we have successfully set up the name node

Name node running

All the playbooks will be in GitHub attached at the end and the installation file in the drive link attached at the end.😎

Once done with setting up the name node now let's move to set up the data node first create a new folder to setup datanode

In that folder, we need the same files as we required to set up in name node only change which we have to to do that is in hdfs-site.xml file that is shown below

hdfs-sire.xml file

Datanode started

Datanode connected

Guys, here we come to the end of this blog I hope you all like it and found it informative. If have any query feel free to reach me :)

Github link:-https://github.com/guptaadi123/Ansible.git

Drive link:

https://drive.google.com/drive/folders/152YHK4LnRiMk4Kup00GhclbtQje3z39l?usp=sharing
Can we integrate LVM with Hadoop??(making our cluster elastic)
By: Bhavesh Kumawat
Tags: Logical Volume, Hadoop

It is an interesting concept to make storage contributed by the different data nodes be customizable and elastic . That means now it is dependent on the user of the data node to allow how much data will be contributed to the cluster and he/she will be able to increase or decrease the size on the go.

Challenges faced in the contemporary way:-

As we know whenever we share any directory with the cluster it shares whole of the storage which is free in the system. Now consider the data node be a individual system which has contributed its own storage to the cluster and has to do its individual task on daily basis. On one day there is high load on the cluster and user also has to perform the task on the data having high storage but he will not be able to do it . So there is need of limiting the storage that is shared to the cluster. So there is interesting task come in play.

Task Description

🌀 Elasticity Task

🔅Integrating LVM with Hadoop and

providing Elasticity to DataNode Storage

🔅Automating LVM Partition using Python-Script.

Solution of the task in steps:-

Step-1

Adding some new hard disk to the system . To view the detailed description we can use #fdisk -l command.

details of the new hard disk of size 5gb each

Step-2

creating physical partition from the new hard disk with commands and we can confirm the creation with the help of #pvdisplay [partition_name]

#pvcreate /dev/sda

#pvcreate /dev/sdb
Apache Hadoop
By: Koravich Sangkaew
Tags: Apache Hadoop, Hadoop, Big Data

Hadoop Core Component

credit: https://databricks.com/glossary/hadoop

In the services of Hadoop, there are 4 major services in Apache Hadoop.

HDFS: Hadoop Distributed File System is the services the data should be store in the nodes of the Hadoop cluster. Also, storage for input and output of MapReduce jobs Yarn: this service might manage the resource of the Hadoop cluster such as compute the number of resources is needed in the job or planing the schedule of job. MapReduce: MapReduce is the big data processing and also is the main function of the Hadoop cluster to compute the job. For example, count all words in the text. It is very useful in a data science job

Algorithm to count the word

4. Common: is the service that supports other modules of Hadoop

To sum up, Apache Hadoop might use for Big data processing by parallelised computing. Furthermore, it would store the data on the distributed node of the Hadoop cluster.
Configure HDFS Cluster Using Ansible
By: Rohit Raut
Tags: Ansible, Hadoop, Hdfs, Ansible Playbook, Configuration Management

Hadoop

Ansible

Please go through the below article, In this article, I have explained ansible installation, architecture, ansible RAL, Ad-hoc command, and Playbook concepts.

What is HDFS Cluster?

A Hadoop cluster is a special type of cluster designed specifically for storing large amounts of unstructured data in a distributed computing environment and also known as the Hadoop Distributed File System. Such clusters run Hadoop’s open-source distributed processing software on low-cost commodity computers. Typically one machine in the cluster is designated as the NameNode and other Nodes are DataNode which share their storage using the network. HDFS cluster is also known as Shared Nothing System the only shared thing is storage over the network.

In this article, we are going to configure Hadoop Cluster Using Automation Tool Ansible.

Ansible Configuration File

Inventory file

Folder SetUP:

here, I have stored all conf files in the template folder.

Variables files:

Let’s get Started

Steps

Configure NameNode Configure DataNodes Configure Client

Step 1: Configure NameNode

Step 2: Configure DataNodes

Step 3: Configure Client

To get the report of HDFS cluster run

hadoop dfsadmin -report

Congratulations, we have successfully configured the HDFS cluster using the Ansible playbook.

GitHub Link: https://github.com/rohitraut3366/Configure-Hadoop-version-1-HDFS-Cluster-Using-Ansible-Playbook.git

Hope This will Help you all.

Thank You For Reading!!😇😇😇

Connect with me On LinkedIn for queries.
👉🏻 Configure Hadoop Using Ansible Playbook
Tags: Ansible, Hadoop, Ansible Playbook, Followers, Learning And Development

Hadoop ❤️ Ansible

Hello folks 🙋🏻‍♂️,

I hope you all are doing well…I am back with one another interesting article. In this article i am going to write a playbook to automate our hadoop cluster, sounds cool huh , let get our hands dirty ✍️

🔰 Prerequisites :

Three Virtual Machines (Controller node, master node, slave node)

hadoop software & jdk in the controller nodes local system

Good network connectivity

🔰 What is ansible ?

Ansible is an open-source software provisioning, configuration management, and application-deployment tool enabling infrastructure as code. It runs on many Unix-like systems, and can configure both Unix-like systems as well as Microsoft Windows. It includes its own declarative language to describe system configuration. Ansible was written by Michael DeHaan and acquired by Red Hat in 2015. Ansible is agentless, temporarily connecting remotely via SSH or Windows Remote Management (allowing remote PowerShell execution) to do its tasks.

So basically, ansible is one of the automation tool, which makes configuration management very easy and time efficient too.

🔰 What is ansible playbook ?

Playbooks are the files where Ansible code is written. Playbooks are written in YAML format. YAML stands for Yet Another Markup Language. Playbooks are one of the core features of Ansible and tell Ansible what to execute. They are like a to-do list for Ansible that contains a list of tasks.

Playbooks contain the steps which the user wants to execute on a particular machine. Playbooks are run sequentially. Playbooks are the building blocks for all the use cases of Ansible.

🔰 Steps to configure ansible playbook for hadoop

#1 — SOFTWARE CONFIGURATION ON TARGET NODES

STEP I : UPDATE INVENTORY WITH MASTER IP & SLAVE IP

inventory.txt

STEP II : COPY & INSTALL SOFTWARE IN BOTH THE MASTER & SLAVE

PLAYBBOK CODE

!- COMMANDS

# ansible-playbook hadoop.yml : to run ansible playbook

# java -version : to check installed java version

# hadoop version: to check installed hadoop version

MASTER & SLAVE NODE -HADOOP & JDK INSTALLED SUCCESSFULLY

As you can after running the playbook , hadoop & jdk software’s are successfully installed on the target nodes.
Meet Hadoop
By: Shraddha Bag
Tags: Hadoop, Introduction To Hadoop, Web Development, Programming, Integration

Doug Cutting and Mike Cafarella started to work on a project which was unnamed back then.They were actually working on a project called ‘Nutch’ .They were trying to build an open source web search engine.

After Google had published its GFS and MapReduce papers,they both had a clear vision and they started to re- create the systems as a part of Nutch.

Soon Yahoo! got interested.Splitting off the distributed computing part of Nutch,they gave birth to what we call today as “Hadoop”. Hadoop soon grew into a technology that had a massive computational power over vast datasets and a technology that could truly scale to the web.

Why Hadoop!

The simple reason to why Hadoop is to manage the flood of petabytes of data coming from social media and other sources as such.

Lets check the stats:

The New York Stock Exchange generates about 4–5 terabytes of data per day.

Facebook hosts more than 240 billion photos,growing at 7 petabytes per moth.

The Internet Archive stores around 18.5 petabytes of data .

So,here we understand that the problem ,although it seems too simple,-’to store and analyze data’ is not at all a simple one.

You already might have analyzed the problem and the answer would be RAID as a solution to the above problem.The larger the amount of data,the more costlier and powerful the hardware would get.The hardware cost also gets increased.Since here we are talking about storing and analyzing petabytes of data.

Now,what are your chances that the hardware stack would never fail!A simple way is to use Redundant array of disks and in case of failure we have another copy ready.

Hadoop has its own way to handle this problem — — yes you guessed its right,its Hadoop Distributed Filesystem(HDFS).

In January 2008,Hadoop had made its own top-level at Apache,confirming its success and its diverse active community.Today Hadoop is widely used in mainstream enterprises.Hadoop’s role as a general -purpose storage and analysis platform for big data has been recognized by the industry.

EMC,IBM.Microsoft and oracle provide support for Hadoop in their available services.Specialized Hadoop companies like Cloudera,Hortonworks and Map R started providing Hadoop solutions.

A journey we would follow to get into Hadoop Architecture.

I)Core Hadoop.

II)Hadoop Ecosystems.

III)Hadoop Case Studies.

The detailed study of Hadoop includes the following:

i)Introduction to MapReduce.

ii)Hadoop filesystems-HDFS.

iii)YARN,Hadoop’s cluster resource management system.

iv)I/O building blocks in Hadoop:data-integrity,compression,serialization and file-based data structures.

v)Developing MapReduce application.

vi)Implementation of MapReduce.

viii)Administration of Hadoop cluster.

ix)Data ingestion into Hadoop:Technologies like Flume and Sqoop

x)Pig,Hive,Crunch,Spark,HBase and Zookeeper.

Fasten your seat belts,we will soon go on to the details and practical knowledge sessions as well.

Reference:Hadoop-The Definitive Guide by Tom White.
Cloud Hadoop 1.4 버전 신규 릴리즈 알아보기
By: Myeongsu Kim
Tags: Ambari, Hadoop, Apache Ranger, Kerberos, Cloud Computing

Learn more. Medium is an open platform where 170 million readers come to find insightful and dynamic thinking. Here, expert and undiscovered voices alike dive into the heart of any topic and bring new ideas to the surface. Learn more

Make Medium yours. Follow the writers, publications, and topics that matter to you, and you’ll see them on your homepage and in your inbox. Explore
Steps to Configure Hadoop and start cluster services using Ansible Playbook
Tags: Ansible, Hadoop, Arth, Vimal Daga, Righteducation

After That we will Create our inventory file which contains IP address of Datanode and Namenode,

Now, we will create Ansible configuration file ansible.cfg ,

After that we will create Ansible Playbook to configure NameNode or MasterNode

- hosts: NameNode

vars:

jdk: jdk-8u171-linux-x64.rpm

hadoop: hadoop-1.2.1–1.x86_64.rpm tasks:

— name: “Copy jdk file”

copy:

dest: “/root/”

src: “/home/ec2-user/{{ jdk }}” - name: “Copy hadoop file”

copy:

dest: “/root/”

src: “/home/ec2-user/{{ hadoop }}” - name: “Install jdk”

command: “rpm -ivh {{ jdk }}” - name: “Install hadoop”

command: “rpm -ivh {{ hadoop }} — force” - name: “Directory Creation”

file:

path: “/root/namenode”

state: directory - name: “Copy core file”

copy:

dest: “/etc/hadoop”

src: “/ansibleWS/Namenode/core-site.xml” - name: “Copy hdfs file”

copy:

dest: “/etc/hadoop”

src: “/ansibleWS/Namenode/hdfs-site.xml” - name: “Formating NameNode”

shell: echo Y | hadoop namenode -format - name: “Start Hadoop Service”

command: “hadoop-daemon.sh start namenode"

After, our NameNode playbook is made we will run using following command:

$ansible-playbook hadoopNN.yml

Valla! Our namenode is configured now its time to make a playbook for DataNode, Let’s do that,

- hosts: DataNode

vars:

jdk: jdk-8u171-linux-x64.rpm

hadoop: hadoop-1.2.1–1.x86_64.rpm tasks:

— name: “Copy jdk file”

copy:

dest: “/root/”

src: “/home/ec2-user/{{ jdk }}” - name: “Copy hadoop file”

copy:

dest: “/root/”

src: “/home/ec2-user/{{ hadoop }}” - name: “Install jdk”

command: “rpm -ivh {{ jdk }}”

ignore_errors: yes - name: “Install hadoop”

command: “rpm -ivh {{ hadoop }} — force”

ignore_errors: yes - name: “Directory Creation”

file:

path: “/root/datanode”

state: directory - name: “Copy core file”

copy:

dest: “/etc/hadoop”

src: “/ansibleWS/Datanode/core-site.xml” - name: “Copy hdfs file”

copy:

dest: “/etc/hadoop”

src: “/ansibleWS/Datanode/hdfs-site.xml” - name: “Start Datanode Service”

command: “hadoop-daemon.sh start datanode”

Running the ansible DataNode playbook:

$ansible-playbook hadoopDN.yml

Hurrahh!! Now our Data node is also configured, let’s see our DataNode is connected to NameNode or not:

$hadoop dfsadmin -report

Finally, We can see that we are able to our Hadoop Cluster is Ready with Automation.
Set up a Hadoop Cluster using Ansible
By: Gaius Reji
Tags: Ansible, Hadoop, Python, Automation, Configuration Management

Creating Playbook

Create a directory as your workspace, for example mkdir /hadoopws

Inside this workspace, create a playbook (extension .yml), for example,

vim hadoop.yml .

Our first line will have the hosts keyword which takes the label of the host-group on which you wish to perform the tasks mentioned. Let’s start by installing the JDK and Hadoop installation files on both Name Node and Data Node.

Copying Installation files for JDK and Hadoop

In my case, I copied the JDK and Hadoop installation files saved on my Controller Node into the Managed Nodes, but you can also use other ansible modules like get_url to directly download the files from a given URL.

Here I use loop and built in variable item to copy multiple files.

Be sure that you download the JDK supported by the Hadoop version you install. Refer here to check which versions are compatible.

You can run your playbook using ansible-playbook <playbook name> to see if everything is working fine. If correct, your Data Node and Name Node should have both the files saved in the destination that you gave.

Installing Java and Hadoop and stopping Firewall

We can use the yum module to install a package with the path as the name attribute, but we need to enter the --force option for the Hadoop installation which is why we can just use the command module.

We’ll also stop the firewall service on both nodes so that they connect to each other when we start the namenode and datanode.

Configuring Name Node

In the name node, we take input for our directory name using the vars_prompt module. Our tasks consists of,

- file module to create the directory

- lineinfile module to enter the configuration lines in the hdfs-site.xml and core-site.xml files in the location /etc/hadoop/. We use the groups[‘namenode’][0] variable to fetch the IP of the namenode from the Inventory file. This is possible because we labelled our name node IP as ‘namenode’.

- We finally format and start the name node using the command module with commands, hadoop namenode -format -force and hadoop-daemon.sh start namenode .

Configuring Data Node

We perform the same operations as the Name Node on the Data Node, except that we use a different variable name for directory, and our hdfs-site.xml changes a bit too. After this we can directly start the datanode.
Devops Solutions And Service Provider Company
By: Hridi Shah
Tags: Hadoop, Cloudcomputing, Big Data Services, DevOps, Data Analytics

We can work with your business to advance your DevOps workflow so that your organisation can begin to use the petabytes of data you are collecting. We understand that businesses want reliable, cloud-based applications which will be fast, readily available, simple to use and bug-free. By bridging the gap between software development and IT operations, we will help your business provide a stable and reliable information infrastructure.

Source control

DevOps is a revolutionary way to manage software and efficiently maintain a high level of data security. Through effective source control, we can assist your business in the management of everything from computer programs and large websites, to document changes and collections of information.

Build automation

An important aspect of ensuring your business provides continuous delivery of a system is the incorporation of a build automation stage. This stage sets out how new features are integrated into the source code of your systems. Through automated building and testing, we will ensure any data services work correctly at a technical level.

Repo management

By providing effective repo management we can improve the stability of your systems and reduce the reliance on external repositories. There are multiple benefits including the saving of time and bandwidth, improved performance, increased control and the provision of a central storage area

Test automation

We can provide test automation services to strengthen the incorporation of new digital technologies, which are continuing to grow at a rapid pace. Using our advanced test automation practices we will reduce the time your company needs to spend on manual tasks, increase process speed and provide timely feedback.

Deployment

We can use our vast experience to effectively manage your deployment processes. Our team will provide everything from workflow visualisation tools to business process maps so that everyone on the business understands the intricacies of system management and deployment.
How to restart Ambari datanode on failure: Automation
By: Sourabh Kalal
Tags: Ambari, Hadoop, Datanodes, Restart, Automation

There are so many times where you need to restart the dead datanode immediately after some failure.

Here is a script which will help you to do the same.
Ansible and hadoop!
Tags: Ansible, Hadoop, Arth, Vimal Daga, Linuxworld

Ansible-

Ansible is an open source IT configuration management (CM) and automation platform, provided by Red Hat. It uses human-readable YAML templates so that users can program repetitive tasks to occur automatically, without learning an advanced language.

Hadoop-

Hadoop is an open-source software framework for storing data and running applications on clusters of commodity hardware. It provides massive storage for any kind of data, enormous processing power and the ability to handle virtually limitless concurrent tasks or jobs.

Task 11.1

Creating hadoop cluster using ansible playbook!

Steps to follow-

1 Check prerequisites-

This includes installing ansible using (pip3 install ansible) and adding IP’s of namenode and datanode to the inventory. Checking the connection using ansible ping command.

2 Creating playbook-

You have to create 2 playbooks, one for datanode and one for namenode, the playbooks are basically the same for both the nodes.

This is the basic playbook for both nodes just you need to change the path for different nodes as in the creating a folder using the root storage, basically replace namenode with datanode in the playbook.

3 Running the playbook-

Command is “ansible-playbook <filename.yml>”

ran successfully

Similarly playbook of datanode ran successfully.

4 Double check!

done!

HENCE HADOOP CLUSTER IS CONFIGURED USING ANSIBLE PLAYBOOK!
Hadoop Cluster Automation with Ansible!!
By: Priyanka Gavali
Tags: Ansible, Hadoop, Hdfs

Hadoop Cluster Automation with Ansible!!

What is Ansible❓❓

Ansible is an open-source automation tool, or platform by RED HAT used for IT tasks such as configuration management, application deployment, intraservice orchestration, and provisioning.It aims to provide large productivity gains to a wide variety of automation challenges. This tool is very simple to use yet powerful enough to automate complex multi-tier IT application environments!!

What is Hadoop❓❓

Apache Hadoop is a collection of open-source software utilities that facilitates using a network of many computers to solve problems involving massive amounts of data and computation. It provides a software framework for distributed storage and processing of big data using the MapReduce programming model.It provides massive storage for any kind of data, enormous processing power and the ability to handle virtually limitless concurrent tasks or jobs.

1.NameNode❕❕

The NameNode is the centerpiece of an HDFS file system in Hadoop. It keeps the directory tree of all files in the file system, and tracks where across the cluster the file data is kept. It does not store the data of these files itself.The NameNode is a Single Point of Failure for the HDFS Cluster

2.DataNode❕❕

DataNodes store data in a Hadoop cluster and is the name of the daemon that manages the data. File data is replicated on multiple DataNodes for reliability and so that localized computation can be executed near the data. Within a cluster, DataNodes should be uniform

3.Client❕❕

Client in Hadoop refers to the Interface used to communicate with the Hadoop Filesystem. There are different type of Clients available with Hadoop to perform different tasks. The basic filesystem client hdfs dfs is used to connect to a Hadoop Filesystem and perform basic file related tasks.

How Ansible Works❓❓

Ansible works by connecting to your nodes and pushing out small programs, called “Ansible modules” to them. … Ansible then executes these modules , and removes them when finished. Library of modules can reside on any machine, and there are no servers, daemons, or databases required.Ansible has its playbook concept to carry out multiple management related tasks.An Ansible playbook contains one or multiple plays, each of which define the work to be done for a configuration on a managed server. Ansible playbooks are written in YAML.

💢Task Completion💢

⏩ Here ,at first there is no any installation of Hadoop Software on namenode ,datanode and client :

⏩ Run the NameNode Playbook by command:

ansible-playbook <name.yml>

⏩ Now,check the master service is started or not at NameNode by Command:

jps

⏩ Run the datanode playbook:

⏩ Check admin report at DataNode by Command:

hadoop dfsadmin -report

⏩ Run the Client Playbook:

⏩ Check file report at Client by command:

hadoop fs -ls /

⏩ Browse the WebUI of Hadoop Cluster :

hdfs://<masterip>:50070

Whole Hadoop Cluster Set up is done Using Ansible Automation Successfully✨✨

🔰Code Link:
Internal & external tables in Hadoop- HIVE (the big data island)
By: Amandeep Modgil
Tags: Bigdata, Hadoop, Hive

What is the difference between internal & external table in Hadoop-HIVE and which one should I use? you might have the above question in your mind too…. the following little tip will help you sort out some of those anomalies.

==============================================================

External table in HIVE

(stores data on HDFS)

External table stores files on the HDFS server but tables are not linked to the source file completely.(I have explained below what I meant by completely)

If you delete an external table the file still remains on the HDFS server.

The file and the table link is there but read only.

As an example if you create an external table called “amandeep_test” in HIVE using HIVE-QL and link the table to file “flat_file.txt”, then deleting “amandeep_test” from HIVE will not delete “flat_file.txt” from HDFS.

External table files are accessible to anyone who has access to HDFS file structure and therefore security needs to be managed at the HDFS file/folder level.

Meta data is maintained on master node and deleting an external table from HIVE, only deletes the metadata not the data/file.

Use external table if you:

Want to manage the data outside HIVE e.g. you are planning to use an ETL tool to load/merge data files etc.

Want to load the latest information to the table but still want to retain old dataset in a file on HDFS for regulatory/legal purposes.

Are not planning to create a table from another table schema e.g. Create table1 as (Select * from table2)

==============================================================

Internal table in HIVE

(stores data on HDFS but in a kind of restricted area)

Stored in a directory based on settings in the following file: hive.metastore.warehouse.dir by default internal tables are stored in the following directory “ /user/hive/warehouse” you can change it by updating the location in the config file mentioned above. e.g. in the following screen shot internal table “ tbl_batting“ is stored as a file on HDFS in warehouse folder

Internal table file security is controlled solely via HIVE. E.g. tbl_batting can only be accessed via HDFS internal account. (refer screenshot below)

r = read

w = write

x = execute

Please note the “user & group” name who can access the file and the security flags next to it in the screenshot below:

Deleting the table deletes the metadata & data from masternode and HDFS respectively

Security needs to be managed within HIVE, probably at the schema level (depends on organisation to organisation). HDFS security is out of scope in this case.

Use internal table if you:

Want to store the data temporary.

Want to use HIVE to manage the lifecycle of tables and data.

There are few other scenarios to consider when modelling Big data platforms, please message me if you may have any question.

Thanks,

Amandeep Modgil
Devops Companies in UK
By: Hridi Shah
Tags: Datamanagement, Hadoop, Bigdata, DevOps, Dataanalytics

We can work with your business to advance your DevOps workflow so that your organisation can begin to use the petabytes of data you are collecting. We understand that businesses want reliable, cloud-based applications which will be fast, readily available, simple to use and bug-free. By bridging the gap between software development and IT operations, we will help your business provide a stable and reliable information infrastructure.

Source control

DevOps is a revolutionary way to manage software and efficiently maintain a high level of data security. Through effective source control, we can assist your business in the management of everything from computer programs and large websites, to document changes and collections of information.

Build automation

An important aspect of ensuring your business provides continuous delivery of a system is the incorporation of a build automation stage. This stage sets out how new features are integrated into the source code of your systems. Through automated building and testing, we will ensure any data services work correctly at a technical level.

Repo management

By providing effective repo management we can improve the stability of your systems and reduce the reliance on external repositories. There are multiple benefits including the saving of time and bandwidth, improved performance, increased control and the provision of a central storage area

Test automation

We can provide test automation services to strengthen the incorporation of new digital technologies, which are continuing to grow at a rapid pace. Using our advanced test automation practices we will reduce the time your company needs to spend on manual tasks, increase process speed and provide timely feedback.

Deployment

We can use our vast experience to effectively manage your deployment processes. Our team will provide everything from workflow visualisation tools to business process maps so that everyone on the business understands the intricacies of system management and deployment.
Managed Analytics Services
By: Hridi Shah
Tags: Hadoop, Data Visualization, Big Data Analytics, DevOps, Cloud Computing

What are Managed Analytics Services?

Managed Analytics is the end to end business analytical solution from the collection, measurement, and analysis of data gathered from various sources to provide better insights to the enterprise.

Why do you need Managed Analytics Services?

Oftentimes enterprise faces a lot of challenges in terms of handling high volume data, building the right pool of talented teams which could be complex, costly and time-consuming. Managed analytics services will provide you a perfect mix of skill sets, tools, expertise, and experience. Managed analytics services also help companies move from CapEx to a predictable Opex cost model.

Predictability makes it easy for business users to justify business cases, and decision-makers approve managed analytic budgets. As services become more results-oriented, costs can be attributed to measured business value.

Over time, companies have introduced a series of business intelligence, analytics, and self-reporting tools that are often motivated by company-level considerations.

However, in due course of time, several unique challenges have made their presence felt, making the need for a qualified & proven arm of Analytics professionals, namely:

Departmental silos that put enterprise-wide views at risk and result in IT assets across the enterprise being leveraged only in small proportions

Low Business and IT collaboration, which makes delivering change requests for the BI (Business Intelligence, Data Warehouse) applications within SLA limits challenging

Organizational dependency on a few critical people for application knowledge, rendering the delivery of mega-projects that beat expectations, a herculean task

Time-intensive migration of platforms for consolidation or upgradation, for even a simple rationalized subset of reports

Why choose cloudaeon for Managed Analytics Services?

No upfront analytics investments in time and resources.

The reduced total cost of ownership.

Faster and simpler data access and analysis.

Easy Management, error-free and quick analysis

Accelerated and Accurate Business Insights

Predictable Business Outcomes

Leverage Big Data Technologies on Cloud

Engagement Model:
Automating Hadoop Cluster Setup Using Ansible
By: Sarthak Srivastava
Tags: Ansible, Hadoop, Automation, DevOps, Ansible Playbook

What is Ansible?

Ansible is an open-source automation tool, or platform, used for IT tasks such as configuration management, application deployment, intraservice orchestration, and provisioning. Automation is crucial these days, with IT environments that are too complex and often need to scale too quickly for system administrators and developers to keep up if they had to do everything manually. Automation simplifies complex tasks, not just making developers’ jobs more manageable but allowing them to focus attention on other tasks that add value to an organization.

What is Hadoop ?

The Hadoop Distributed File System (HDFS) is a distributed file system designed to run on commodity hardware. It has many similarities with existing distributed file systems. However, the differences from other distributed file systems are significant. HDFS is highly fault-tolerant and is designed to be deployed on low-cost hardware.

Ansible Architecture :

Ansible Playbooks :

Ordered lists of tasks, saved so you can run those tasks in that order repeatedly. Playbooks are written in YAML format.

Inventory :

A list of managed nodes. An inventory file is also sometimes called a hostfile. Your inventory can specify information like IP address for each managed node.

Control Node:

Any machine with Ansible installed is known as controller node. You can run Ansible commands and playbooks by invoking the ansible or ansible-playbook command from any control node.

Managed Node:

The devices you manage with Ansible. Managed nodes are also sometimes called hosts.

Now lets start setup of Hadoop Cluster using Ansible.

Steps :

Install ansible in controller mode

pip3 install ansible

yum install sshpass#To see version of ansible installed

ansible --version

Ansible Configuration File

vim /etc/ansible/ansible.conf

To check connectivity with all Managed Nodes

ansible all -m ping

PlayBook :

PlayBook for Configration of NameNode

Running NameNode Playbook

ansible-playbook namenode.yml

Checking namenode services has been started in target node

PlayBook for Configration of DataNode

Running DataNode Playbook

(Due to low RAM and CPU resources I am making DataNode same VM that I used for NameNode. For that just stop the NameNode Services and delete the NameNode folder.)

ansible-playbook datanode.yml

PlayBook for Configration of Client

Link to Code :
Automation Of Technologies Using Python
By: Mangesh Ghorpade
Tags: Hadoop, Python, AWS, Docker, Lvm

♦️What is Automation?

If a developer has to repeat a job more than once, then instead of coding the same thing or doing the same process multiple times, they make a small piece of software or a bot that does the job for them. That’s Automation.

♦️These Options include various tasks that can be automated with the push of a single button, and Also Options to Navigate throughout the Automation Menu.

We have built a Python Menu Program which can manage technologies like AWS, Docker, Hadoop, and LVM Partitioning.

♦️What is Hadoop?

Apache Hadoop is an open-source, Java-based, software framework and parallel data processing engine. It enables big data analytics processing tasks to be broken down into smaller tasks that can be performed in parallel by using an algorithm (like the MapReduce algorithm) and distributing them across a Hadoop cluster.

Code to integrate with Hadoop

♦️What is LVM ?

LVM is a tool for logical volume management which includes allocating disks, striping, mirroring and resizing logical volumes.

With LVM, a hard drive or set of hard drives is allocated to one or more physical volumes. LVM physical volumes can be placed on other block devices which might span two or more disks. LVM provides features like the ability to add disk space to a logical volume and its filesystem. LVM helps to provide the Elasticity to the storage Device.

Code to integrate with LVM:-

Code to integrate with AWS:-

♦️ What is DOCKER?

Docker is a set of platform as a service (PaaS) products that use OS-level virtualization to deliver software in packages called containers. Docker has a capability to launch OS in 1 second which is called as docker container. It is highly efficient with low latency and high speed. Here, Containerization is a technology and docker is one of its highly used product.

Code to integrate with Docker:-

In this way, Task is Completed Successfully !!

Hope You Will Find It Usefull!!!
Menu Using Python integrating Hadoop , Aws ,Docker, LVM .
Tags: Hadoop, Python, AWS, Docker, Lvm

Lets see , how we can integrate Hadoop , Aws ,Docker, LVM using Python

Python- It’s often used as a “scripting language” for web applications. This means that it can automate specific series of tasks, making it more efficient. Consequently, Python (and languages like it) is often used in software applications, pages within a web browser, the shells of operating systems and some games.

Menu Program

Code of Menu Program :

Hadoop -Hadoop is an Apache open source framework written in java that allows distributed processing of large datasets across clusters of computers using simple programming models. The Hadoop framework application works in an environment that provides distributed storage and computation across clusters of computers. Hadoop is designed to scale up from single server to thousands of machines, each offering local computation and storage.

Code to integrate with Hadoop :

LVM -LVM is a tool for logical volume management which includes allocating disks, striping, mirroring and resizing logical volumes.With LVM, a hard drive or set of hard drives is allocated to one or more physical volumes. LVM physical volumes can be placed on other block devices which might span two or more disks.The physical volumes are combined into logical volumes, with the exception of the /boot partition. The /boot partition cannot be on a logical volume group because the boot loader cannot read it. If the root ( / ) partition is on a logical volume, create a separate /boot partition which is not a part of a volume group.

Code to integrate with LVM :

AWS -Amazon Web Services(AWS) is a cloud service from Amazon, which provides services in the form of building blocks, these building blocks can be used to create and deploy any type of application in the cloud.These services or building blocks are designed to work with each other, and result in applications that are sophisticated and highly scalable.

Code to integrate with AWS :

Docker - Docker is an open platform for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly. With Docker, you can manage your infrastructure in the same ways you manage your applications. By taking advantage of Docker’s methodologies for shipping, testing, and deploying code quickly, you can significantly reduce the delay between writing code and running it in production.

Code to integrate with Docker:
Hive Array Data Type and its Function
By: Aswadi A.R
Tags: Etl, Arrays, Hadoop, Hive

Picture courtesy by Radicasoftware

So last week when doing Extract-Transform-Load() for my company, I stumbled around a weird data type. It looks like a Set data type that I usually found in python language. By the way, my data was a pipe(|) separated file but name as CSV. Doing an array data type will help me to easily pluck the element/value inside the array and the same time reducing my ETL process.

{0,0,0,0}|{4667,0,0,0,4667,0,4667,0,1105}

{0,0,0,0}|{10000,0,0,0,10000,0,10000,0,1105}|{0,0,0,0}

After creating the hive table using below script, I manage to load the data using string data type. I can’t load directly into hive array data type because you need to use open/close bracket ([]) instead of curly open/close bracket({}). Using load data inpath with character wild card (?), I will be able to load all file that have “etl_” name and end with “.csv”.

create table test_table(

complex_field string,

complex_field1 string)

ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘|’

STORED AS TEXTFILE; load data inpath '/user/aswadi/etl_??????????????_?????.csv' into table test_table;

Next we create the real table for our data that have array data type and insert the data in test_table that we load just now. I chose array<string> instead of array<int> since I encounter a lot of issue when dealing with hive array<int> data type.

create table test_table_array(

complex_field array<string>,

complex_field1 array<string>)

ROW FORMAT DELIMITED FIELDS TERMINATED BY ‘|’ collection items terminated by ‘,’ STORED AS TEXTFILE; insert into table test_table_array select

,array(regexp_replace(complex_field,'[{}]',''))

,array(regexp_replace(complex_field1,'[{}]',''))

from test_table;

Once we load the data into the final table, we can start query the data using normal SQL using column[n], n stand for the number of array.

select

complex_field[0],

complex_field1[0]

from test_table_array limit 10;

hive array datatype query

On top of that we can take advantage of hive array default User Defined Function (UDF) as per below.