Akshay Rai – Dr. Elephant - Self-Serve Performance Tuning for Hadoop and Spark
Channel: Hasgeek TV
Date: 2016-07-31

Slides: http://www.slideshare.net/aksmlore/the-fifth-elephant-2016-selfserve-performance-tuning-for-hadoop-and-spark

Hadoop is a framework that facilitates the distributed storage and processing of large distributed datasets involving a number of components interacting with each other. Because of its large and complex framework, it is important to make sure every component performs optimally. While we can always optimize the underlying hardware resources, network infrastructure, OS, and other components of the stack, only users have control over optimizing the jobs that run on the cluster.

Dr. Elephant is a tool for the users of Hadoop to help them understand, analyse and tune their Hadoop/Spark applications easily, thus improving their productivity and the cluster’s efficiency. It analyzes the Hadoop and Spark jobs using a set of pluggable, configurable, rule-based heuristics that provide insights on how a job performed, and then uses the results to make suggestions about how to tune the job to make it perform more efficiently.