The charm of Apache Pig
By: Prasadi Abeywardana
Tags: Hadoop, Big Data, Mapreduce, Apache Pig, Scripting

The charm of Apache Pig

A big data tool not to miss

Apache Pig is a core piece of technology in the Hadoop eco-system. The objective of this article is to discuss how Apache Pig becomes prominent among rest of the Hadoop tech tools and why and when someone should utilize Pig for their big data tasks.

Anyone with a reasonable understanding of Hadoop knows that it is capable of handling large volumes of data in a distributed manner. Hadoop’s programming model called MapReduce enables processing large datasets in parallel by splitting a job into a set of map tasks and reduce tasks. MapReduce is the underlying low-level programming model and these jobs can be implemented using languages like Java and Python.

Consider a simple word count task that we want to achieve via Hadoop. The intention of this job is to count the number of occurrences of each word in a given input set. Following code snippet shows how this can be achieved using a mapper class and a reducer class written using Java. This is an example provided in https://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduceTutorial.html#Source_Code

import java.io.IOException;

import java.util.StringTokenizer; import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IntWritable;

import org.apache.hadoop.io.Text;

import org.apache.hadoop.mapreduce.Job;

import org.apache.hadoop.mapreduce.Mapper;

import org.apache.hadoop.mapreduce.Reducer;

import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;

import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat; public class WordCount { public static class TokenizerMapper

extends Mapper<Object, Text, Text, IntWritable>{ private final static IntWritable one = new IntWritable(1);

private Text word = new Text(); public void map(Object key, Text value, Context context

) throws IOException, InterruptedException {

StringTokenizer itr = new StringTokenizer(value.toString());

while (itr.hasMoreTokens()) {

word.set(itr.nextToken());

context.write(word, one);

}

}

} public static class IntSumReducer

extends Reducer<Text,IntWritable,Text,IntWritable> {

private IntWritable result = new IntWritable(); public void reduce(Text key, Iterable<IntWritable> values,

Context context

) throws IOException, InterruptedException {

int sum = 0;

for (IntWritable val : values) {

sum += val.get();

}

result.set(sum);

context.write(key, result);

}

} public static void main(String[] args) throws Exception {

Configuration conf = new Configuration();

Job job = Job.getInstance(conf, "word count");

job.setJarByClass(WordCount.class);

job.setMapperClass(TokenizerMapper.class);

job.setCombinerClass(IntSumReducer.class);

job.setReducerClass(IntSumReducer.class);

job.setOutputKeyClass(Text.class);

job.setOutputValueClass(IntWritable.class);

FileInputFormat.addInputPath(job, new Path(args[0]));

FileOutputFormat.setOutputPath(job, new Path(args[1]));

System.exit(job.waitForCompletion(true) ? 0 : 1);

}

}

OK, so this code snippet consists of 61 lines of code. To write something like this, you must clearly understand the fundamental concepts and theories of MapReduce. And, you should know the set of libraries or packages provided in your preferred language and how they should be put together. Mappers, reducers, and other utility methods must be called or combined explicitly into a pipeline to achieve the overall outcome (main method in this example) Out of all, this piece of code has to be compiled and deployed. Sounds a little complex right, especially if you do not have a MapReduce or Java background?

When I wrote my first MapReduce program using Java, I asked this question myself — ‘isn’t there any easier way of achieving this?’. The answer is ‘Yes, there is, and that is with Apache Pig’.

Apache Pig is a high-level platform for creating programs that run on Hadoop. The language for this platform is called Pig Latin.

Pig Latin is a SQL like scripting language, that abstracts the programming concepts of MapReduce. If you are familiar with basic concepts of SQL, putting a MapReduce job in place using Pig Latin is a task of just a few minutes. It converts the underlying complex concepts of MapReduce into a very high-level programming model.

Now let’s see how the above word count task we did with Java MapReduce can be achieved using Pig Latin (https://en.wikipedia.org/wiki/Apache_Pig).

input_lines = LOAD '/tmp/word.txt' AS (line:chararray);



words = FOREACH input_lines GENERATE FLATTEN(TOKENIZE(line)) AS word;



filtered_words = FILTER words BY word MATCHES '\\w+';



word_groups = GROUP filtered_words BY word;



word_count = FOREACH word_groups GENERATE COUNT(filtered_words) AS count, group AS word; ordered_word_count = ORDER word_count BY count DESC; STORE ordered_word_count INTO '/tmp/results.txt';

With Pig Latin, we have reduced the previous 61 lines to 7 lines. And the same task is achieved by manipulating a couple of keywords. And a piece of code like this is easy to implement as well as it is easy to be understood by someone else. So, it is obvious that Apache Pig reduces the learning and development curve of MapReduce. Understanding the above 7 lines of code is the challenge I am going to leave for you. If you are interested, you can start learning basics of Pig Latin here: https://pig.apache.org/docs/r0.17.0/basic.html

Following table provides a summarized comparison of Apache Pig and MapReduce based on the above example.