Yelp/mrjob — Issue #377: Pig Support for Mrjob
Author: sshivaji
Created: 2012-02-24 01:32:13+00:00
State: open

Use case for Pig:

Pig and Hive are known high level languages over map reduce jobs. While Hive emulates a SQL dialect, Pig has a more flexible language. There is a learning curve to understanding pig latin but as jobs become complex, it might be more useful to design pig scripts rather than actual code.

Why Pig is useful with Mrjob:

In their latest versions and the one on EMR, you can actually write out the UDF for Pig in python! You only have to deal with python and pig scripts to get the job to run on EMR.

I wrote a custom UDF in python and a locally tested pig script. I then bootstrapped EMR to install a jython jar and some extra python libraries (a 2nd jar containing these taken from the jython project). After that my pig scripts were able to run. I used Amazon's elastic-mapreduce tool to run the pig script. It ran fine. Mrjob is far more convenient to use than the ruby based elastic-mapreduce tool provided by Amazon.

I have pig support with my fork of mrjob.

What's supported:
1. Running pig jobs on emr.
2. Pig parameter substitution.
3. Running pig jobs locally (if a pig executable exists locally).

What's not well supported:
1. Progress reporting. This is more complicated as instead of one map reduce job, you can have 5 or 10 if you set the parallel keyword in pig. In fact, its optimal to set the parallel keyword.
2. Pig log files in case a job fails (need to add these files to the regex path). The pig files are stored in /var/lib/hadoop .. instead of /var/log/hadoop.
3. Bootstrap scripts for jython jars and the UDF, currently one has to manually specify --bootstrap-action to upload the jython jar and the UDF. This can be improved.
4. Pig is installed in the first step of any pig job. This is annoying if one is using a persistent job flow. Persistent job flows still work, but arguably will be slightly slower as the pig jars are installed every time a job is submitted.

Will initiate a pull request with my current version.