Hadoop: Apache's Open Source Implementation of Google's Map/Reduce Framework 2/4 - Defcon 17
Channel: HackedExistence
Date: 2012-07-13

This is part 2 of the speech "Hadoop: Apache's Open Source Implementation of Google's Map/Reduce Framework" given at Defcon 17 in Las Vegas by the Hacked Existence Team.

This video starts by looking at the Master Node and the Job Tracker.  Then we look at the Distributed Cache and the Streaming Interface as two methods of getting input/output into and out of the cloud.  Then we look at the Hadoop Distributed File System (HDFS), Apache's equivalent of Google's GFS, and the File System that is used by the Hadoop Cloud.

Jumping into Databases, we crack open HBase, a non-relational three-dimensional database that is used with Hadoop to store/retrieve data.

We then look at the options provided by Amazon's EC2 for throwing up your own elastic Hadoop Cloud.

We then jump into the code side of the speech by looking at the mapper/reducer we wrote for the Netflix Prize data set, a 2 gig data set provided by Netflix for a competition to write a recommendation algorithm for movies.

Project page and source code from this speech can be found here: http://hackedexistence.com/project-hadoop.html